Messages in this thread | | | From | Song Liu <> | Subject | Re: [PATCH v6 3/4] perf-stat: enable counting events for BPF programs | Date | Tue, 29 Dec 2020 19:11:12 +0000 |
| |
> On Dec 29, 2020, at 10:48 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > Em Tue, Dec 29, 2020 at 06:42:18PM +0000, Song Liu escreveu: >> >> >>> On Dec 29, 2020, at 7:15 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote: >>> >>> Em Mon, Dec 28, 2020 at 11:43:25PM +0000, Song Liu escreveu: >>>> >>>> >>>>> On Dec 28, 2020, at 12:11 PM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote: >>>>> >>>>> Em Mon, Dec 28, 2020 at 09:40:53AM -0800, Song Liu escreveu: >>>>>> Introduce perf-stat -b option, which counts events for BPF programs, like: >>>>>> >>>>>> [root@localhost ~]# ~/perf stat -e ref-cycles,cycles -b 254 -I 1000 >>>>>> 1.487903822 115,200 ref-cycles >>>>>> 1.487903822 86,012 cycles >>>>>> 2.489147029 80,560 ref-cycles >>>>>> 2.489147029 73,784 cycles >>>>>> 3.490341825 60,720 ref-cycles >>>>>> 3.490341825 37,797 cycles >>>>>> 4.491540887 37,120 ref-cycles >>>>>> 4.491540887 31,963 cycles >>>>>> >>>>>> The example above counts cycles and ref-cycles of BPF program of id 254. >>>>>> This is similar to bpftool-prog-profile command, but more flexible. >>>>>> >>>>>> perf-stat -b creates per-cpu perf_event and loads fentry/fexit BPF >>>>>> programs (monitor-progs) to the target BPF program (target-prog). The >>>>>> monitor-progs read perf_event before and after the target-prog, and >>>>>> aggregate the difference in a BPF map. Then the user space reads data >>>>>> from these maps. >>>>>> >>>>>> A new struct bpf_counter is introduced to provide common interface that >>>>>> uses BPF programs/maps to count perf events. >>>>> >>>>> Segfaulting here: >>>>> >>>>> [root@five ~]# bpftool prog | grep tracepoint >>>>> 110: tracepoint name syscall_unaugme tag 57cd311f2e27366b gpl >>>>> 111: tracepoint name sys_enter_conne tag 3555418ac9476139 gpl >>>>> 112: tracepoint name sys_enter_sendt tag bc7fcadbaf7b8145 gpl >>>>> 113: tracepoint name sys_enter_open tag 0e59c3ac2bea5280 gpl >>>>> 114: tracepoint name sys_enter_opena tag 0baf443610f59837 gpl >>>>> 115: tracepoint name sys_enter_renam tag 24664e4aca62d7fa gpl >>>>> 116: tracepoint name sys_enter_renam tag 20093e51a8634ebb gpl >>>>> 117: tracepoint name sys_enter tag 0bc3fc9d11754ba1 gpl >>>>> 118: tracepoint name sys_exit tag 29c7ae234d79bd5c gpl >>>>> [root@five ~]# >>>>> [root@five ~]# gdb perf >>>>> GNU gdb (GDB) Fedora 10.1-2.fc33 >>>>> Reading symbols from perf... >>>>> (gdb) run stat -e instructions,cycles -b 113 -I 1000 >>>>> Starting program: /root/bin/perf stat -e instructions,cycles -b 113 -I 1000 >>>>> [Thread debugging using libthread_db enabled] >>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>> libbpf: elf: skipping unrecognized data section(9) .eh_frame >>>>> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame >>>>> libbpf: elf: skipping unrecognized data section(9) .eh_frame >>>>> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame >>>>> >>>>> Program received signal SIGSEGV, Segmentation fault. >>>>> 0x000000000058d55b in bpf_program_profiler__read (evsel=0xc612c0) at util/bpf_counter.c:217 >>>>> 217 reading_map_fd = bpf_map__fd(skel->maps.accum_readings); >>>>> (gdb) bt >>>>> #0 0x000000000058d55b in bpf_program_profiler__read (evsel=0xc612c0) at util/bpf_counter.c:217 >>>>> #1 0x0000000000000000 in ?? () >>>>> (gdb) >>>>> >>>>> [acme@five perf]$ clang -v |& head -2 >>>>> clang version 11.0.0 (Fedora 11.0.0-2.fc33) >>>>> Target: x86_64-unknown-linux-gnu >>>>> [acme@five perf]$ >>>>> >>>>> Do you need any extra info? >>>> >>>> Hmm... I am not able to reproduce this. I am trying to setup an environment similar >>>> to fc33 (clang 11, etc.). Does this segfault every time, and on all programs? >>> >>> I'll try it with a BPF proggie attached to a kprobes, but here is >>> something else I noticed: >>> >>> [root@five perf]# export PYTHONPATH=/tmp/build/perf/python >>> [root@five perf]# tools/perf/python/twatch.py >>> Traceback (most recent call last): >>> File "/home/acme/git/perf/tools/perf/python/twatch.py", line 9, in <module> >>> import perf >>> ImportError: /tmp/build/perf/python/perf.cpython-39-x86_64-linux-gnu.so: undefined symbol: bpf_counter__destroy >>> [root@five perf]# perf test python >>> 19: 'import perf' in python : FAILED! >>> [root@five perf]# perf test -v python >>> 19: 'import perf' in python : >>> --- start --- >>> test child forked, pid 3198864 >>> python usage test: "echo "import sys ; sys.path.append('/tmp/build/perf/python'); import perf" | '/usr/bin/python3' " >>> Traceback (most recent call last): >>> File "<stdin>", line 1, in <module> >>> ImportError: /tmp/build/perf/python/perf.cpython-39-x86_64-linux-gnu.so: undefined symbol: bpf_counter__destroy >>> test child finished with -1 >>> ---- end ---- >>> 'import perf' in python: FAILED! >>> [root@five perf]# >>> >>> This should be trivial, I hope, just add the new object file to >>> tools/perf/util/python-ext-sources, then do a 'perf test python', if it >>> fails, use 'perf test -v python' to see what is preventing the python >>> binding from loading. >> >> I fixed the undefined bpf_counter__destroy. But this one looks trickier: >> >> 19: 'import perf' in python : >> --- start --- >> test child forked, pid 2714986 >> python usage test: "echo "import sys ; sys.path.append('python'); import perf" | '/bin/python2' " >> Traceback (most recent call last): >> File "<stdin>", line 1, in <module> >> ImportError: XXXXX /tools/perf/python/perf.so: undefined symbol: bpf_map_update_elem >> >> Given I already have: > > I'll check this one to get a patch that at least moves the needle here, > i.e. probably we can leave supporting bpf counters in the python binding > for a later step.
Thanks Arnaldo!
Currently, I have: 1. Fixed issues highlighted by Namhyung; 2. Merged 3/4 and 4/4; 3. NOT found segfault; 4. NOT fixed python import perf.
I don't have good ideas with 3 and 4... Shall I send current code as v7?
Thanks, Song
> > - Arnaldo > >> diff --git i/tools/perf/util/python-ext-sources w/tools/perf/util/python-ext-sources >> index a9d9c142eb7c3..2cac55273eca2 100644 >> --- i/tools/perf/util/python-ext-sources >> +++ w/tools/perf/util/python-ext-sources >> @@ -35,3 +35,6 @@ util/symbol_fprintf.c >> util/units.c >> util/affinity.c >> util/rwsem.c >> +util/bpf_counter.c >> +../lib/bpf/bpf.c >> +../lib/bpf/libbpf.c >> >> >> How should I fix this? >> >> Thanks, >> Song >> >> PS: I still cannot reproduce that segfault... >> >>> >> > > -- > > - Arnaldo
| |