lkml.org 
[lkml]   [2013]   [Oct]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 3/3] perf record: mmap output file

* David Ahern <dsahern@gmail.com> wrote:

> When recording raw_syscalls for the entire system, e.g.,
> perf record -e raw_syscalls:*,sched:sched_switch -a -- sleep 1
>
> you end up with a negative feedback loop as perf itself calls
> write() fairly often. This patch handles the problem by mmap'ing the
> file in chunks of 64M at a time and copies events from the event buffers
> to the file avoiding write system calls.
>
> Before (with write syscall):
>
> perf record -o /tmp/perf.data -e raw_syscalls:*,sched:sched_switch -a -- sleep 1
> [ perf record: Woken up 0 times to write data ]
> [ perf record: Captured and wrote 81.843 MB /tmp/perf.data (~3575786 samples) ]
>
> After (using mmap):
>
> perf record -o /tmp/perf.data -e raw_syscalls:*,sched:sched_switch -a -- sleep 1
> [ perf record: Woken up 31 times to write data ]
> [ perf record: Captured and wrote 8.203 MB /tmp/perf.data (~358388 samples) ]
>
> In addition to perf-trace benefits using mmap lowers the overhead of
> perf-record. For example,
>
> perf stat -i -- perf record -g -o /tmp/perf.data openssl speed aes
>
> showsi a drop in time, CPU cycles, and instructions all drop by more than a
> factor of 3. Jiri also ran a test that showed a big improvement.

Here are some thoughts on how 'perf record' tracing performance could be
further improved:

1)

The use of non-temporal stores (MOVNTQ) to copy the ring-buffer into the
file buffer makes sure the CPU cache is not trashed by the copying - which
is the largest 'collateral damage' copying does.

glibc does not appear to expose non-temporal instructions so it's going to
be architecture dependent - but we could build the copy_user_nocache()
function from the kernel proper (or copy it - we could even simplify it:
knowing that only large and page aligned buffers are going to be copied
with it).

See how tools/perf/bench/mem-mem* does that to be able to measure the
kernel's memcpy() and memset() function performance.

2)

Yet another method would be to avoid the copies altogether via the splice
system-call - see:

git grep splice kernel/trace/

To make splice low-overhead we'd have to introduce a mode to not mmap the
data part of the perf ring-buffer and splice the data straight from the
perf fd into a temporary pipe and over from the pipe into the target file
(or socket).

OTOH non-temporal stores are incredibly simple and memory bandwidth is
plenty on modern systems so I'd certainly try that route first.

Thanks,

Ingo


\
 
 \ /
  Last update: 2013-10-09 08:21    [W:0.107 / U:0.320 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site