Messages in this thread | | | Date | Wed, 9 Jan 2019 16:51:39 +0100 | From | Jiri Olsa <> | Subject | Re: [PATCH v3 0/4] Reduce NUMA related overhead in perf record profiling on large server systems |
| |
On Wed, Jan 09, 2019 at 03:41:25PM +0100, Jiri Olsa wrote: > On Wed, Jan 09, 2019 at 12:19:20PM +0300, Alexey Budankov wrote: > > > > It has been observed that trace reading thread runs on the same hw thread > > most of the time during perf record sampling collection. This scheduling > > layout leads up to 30% profiling overhead in case when some cpu intensive > > workload fully utilizes a large server system with NUMA. Overhead usually > > arises from remote (cross node) HW and memory references that have much > > longer latencies than local ones [1]. > > > > This patch set implements --affinity option that lowers 30% overhead > > completely for serial trace streaming (--affinity=cpu) and from 30% to > > 10% for AIO1 (--aio=1) trace streaming (--affinity=node|cpu). > > See OVERHEAD section below for more details. > > > > Implemented extension provides users with capability to instruct Perf > > tool to bounce trace reading thread's affinity mask between NUMA nodes > > (--affinity=node) or assign the thread to the exact cpu (--affinity=cpu) > > that trace buffer being processed belongs to. > > > > The extension brings improvement in case of full system utilization when > > Perf tool process contends with workload process on cpu cores. In case a > > system has free cores to execute Perf tool process during profiling the > > default system scheduling layout induces the lowest overhead. > > > > The patch set has been validated on BT benchmark from NAS Parallel > > Benchmarks [2] running on dual socket, 44 cores, 88 hw threads Broadwell > > system with kernels v4.4-21-generic (Ubuntu 16.04) and v4.20.0-rc5 > > (tip perf/core). > > > > OVERHEAD: > > BENCH REPORT BASED ELAPSED TIME BASED > > v4.20.0-rc5 > > (tip perf/core): > > > > (current) SERIAL-SYS / BASE : 1.27x (14.37/11.31), 1.29x (15.19/11.69) > > SERIAL-NODE / BASE : 1.15x (13.04/11.31), 1.17x (13.79/11.69) > > SERIAL-CPU / BASE : 1.00x (11.32/11.31), 1.01x (11.89/11.69) > > > > AIO1-SYS / BASE : 1.29x (14.58/11.31), 1.29x (15.26/11.69) > > AIO1-NODE / BASE : 1.08x (12.23/11.31), 1,11x (13.01/11.69) > > AIO1-CPU / BASE : 1.07x (12.14/11.31), 1.08x (12.83/11.69) > > > > v4.4.0-21-generic > > (Ubuntu 16.04 LTS): > > > > (current) SERIAL-SYS / BASE : 1.26x (13.73/10.87), 1.29x (14.69/11.32) > > SERIAL-NODE / BASE : 1.19x (13.02/10.87), 1.23x (14.03/11.32) > > SERIAL-CPU / BASE : 1.03x (11.21/10.87), 1.07x (12.18/11.32) > > > > AIO1-SYS / BASE : 1.26x (13.73/10.87), 1.29x (14.69/11.32) > > AIO1-NODE / BASE : 1.10x (12.04/10.87), 1.15x (13.03/11.32) > > AIO1-CPU / BASE : 1.12x (12.20/10.87), 1.15x (13.09/11.32) > > > > The patch set is generated for acme perf/core repository. > > > > --- > > Alexey Budankov (4): > > perf record: allocate affinity masks > > perf record: bind the AIO user space buffers to nodes > > perf record: apply affinity masks when reading mmap buffers > > perf record: implement --affinity=node|cpu option > > > hi, > can't apply your code on latest Arnaldo's perf/core: > > Applying: perf record: allocate affinity masks > Applying: perf record: bind the AIO user space buffers to nodes > Applying: perf record: apply affinity masks when reading mmap buffers > Applying: perf record: implement --affinity=node|cpu option > error: corrupt patch at line 62 > Patch failed at 0004 perf record: implement --affinity=node|cpu option > Use 'git am --show-current-patch' to see the failed patch > When you have resolved this problem, run "git am --continue". > If you prefer to skip this patch, run "git am --skip" instead. > To restore the original branch and stop patching, run "git am --abort".
hum, when I separate the raw patch and apply it works with no fuzz:
[jolsa@krava perf]$ patch -p3 < /tmp/krava patching file Documentation/perf-record.txt patching file builtin-record.c
this email header caught my eye:
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0
but no idea what's the issue in here ;-)
jirka
| |