lkml.org 
[lkml]   [2022]   [Mar]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH] tracing/user_events: Add eBPF interface for user_event created events
On Wed, Mar 30, 2022 at 12:15 PM Beau Belgrave
<beaub@linux.microsoft.com> wrote:
>
> On Wed, Mar 30, 2022 at 11:22:32AM -0700, Alexei Starovoitov wrote:
> > On Wed, Mar 30, 2022 at 9:34 AM Beau Belgrave <beaub@linux.microsoft.com> wrote:
> > > > >
> > > > > But you are fine with uprobe costs? uprobes appear to be much more costly
> > > > > than a syscall approach on the hardware I've run on.
> >
> > Care to share the numbers?
> > uprobe over USDT is a single trap.
> > Not much slower compared to syscall with kpti.
> >
>
> Sure, these are the numbers we have from a production device.
>
> They are captured via perf via PERF_COUNT_HW_CPU_CYCLES.
> It's running a 20K loop emitting 4 bytes of data out.
> Each 4 byte event time is recorded via perf.
> At the end we have the total time and the max seen.
>
> null numbers represent a 20K loop with just perf start/stop ioctl costs.
>
> null: min=2863, avg=2953, max=30815
> uprobe: min=10994, avg=11376, max=146682

I suspect it's a 3 trap case of uprobe.
USDT is a nop. It's a 1 trap case.

> uevent: min=7043, avg=7320, max=95396
> lttng: min=6270, avg=6508, max=41951
>
> These costs include the data getting into a buffer, so they represent
> what we would see in production vs the trap cost alone. For uprobe this
> means we created a uprobe and attached it via tracefs to get the above
> numbers.
>
> There also seems to be some thinking around this as well from Song Liu.
> Link: https://lore.kernel.org/lkml/20200801084721.1812607-1-songliubraving@fb.com/
>
> From the link:
> 1. User programs are faster. The new selftest added in 5/5, shows that a
> simple uprobe program takes 1400 nanoseconds, while user program only
> takes 300 nanoseconds.


Take a look at Song's code. It's 2 trap case.
The USDT is a half of that. ~700ns.
Compared to 300ns of syscall that difference
could be acceptable.

>
> > > >
> > > > Can we achieve the same/similar performance with sys_bpf(BPF_PROG_RUN)?
> > > >
> > >
> > > I think so, the tough part is how do you let the user-space know which
> > > program is attached to run? In the current code this is done by the BPF
> > > program attaching to the event via perf and we run the one there if
> > > any when data is emitted out via write calls.
> > >
> > > I would want to make sure that operators can decide where the user-space
> > > data goes (perf/ftrace/eBPF) after the code has been written. With the
> > > current code this is done via the tracepoint callbacks that perf/ftrace
> > > hook up when operators enable recording via perf, tracefs, libbpf, etc.
> > >
> > > We have managed code (C#/Java) where we cannot utilize stubs or traps
> > > easily due to code movement. So we are limited in how we can approach
> > > this problem. Having the interface be mmap/write has enabled this
> > > for us, since it's easy to interact with in most languages and gives us
> > > lifetime management of the trace objects between user-space and the
> > > kernel.
> >
> > Then you should probably invest into making USDT work inside
> > java applications instead of reinventing the wheel.
> >
> > As an alternative you can do a dummy write or any other syscall
> > and attach bpf on the kernel side.
> > No kernel changes are necessary.
>
> We only want syscall/tracing overheads for the specific events that are
> hooked. I don't see how we could hook up a dummy write that is unique
> per-event without having a way to know when the event is being traced.

You're adding writev-s to user apps. Keep that writev without
any user_events on the kernel side and pass -1 as FD.
Hook bpf prog to sys_writev and filter by pid.

\
 
 \ /
  Last update: 2022-03-30 22:40    [W:0.122 / U:1.000 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site