lkml.org 
[lkml]   [2017]   [Sep]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRough notes from sys_membarrier() lightning BoF
    Hello!

    Rough notes from our discussion last Thursday. Please reply to the
    group with any needed elaborations or corrections.

    Adding Andy and Michael on CC since this most closely affects their
    architectures. Also adding Dave Watson and Maged Michael because
    the preferred approach requires that processes wanting to use the
    lightweight sys_membarrier() do a registration step.

    Thanx, Paul

    ------------------------------------------------------------------------

    Problem:

    1. The current sys_membarrier() introduces an smp_mb() that
    is not otherwise required on powerpc.

    2. The envisioned JIT variant of sys_membarrier() assumes that
    the return-to-user instruction sequence handling any change
    to the usermode instruction stream, and Andy Lutomirski's
    upcoming changes invalidate this assumption. It is believed
    that powerpc has a similar issue.


    Here are diagrams indicating the memory-ordering requirements:

    Scenario 1: Access preceding sys_membarrier() must see changes
    from thread that concurrently switches in.

    ----------------------------------------------------------------

    Scheduler sys_membarrier()
    --------- ----------------

    smp_mb();

    usermode load or store to Y

    /* begin system call */

    sys_membarrier()
    smp_mb();
    Check rq->curr

    rq->curr = new_thread;
    smp_mb(); /* not powerpc! */

    /* return to user */

    usermode load or store to X

    smp_mb();

    ----------------------------------------------------------------

    Due to the fr link from the check of rq->curr to the scheduler's
    write, we need full memory barriers on both sides. However,
    we don't want to lose the powerpc optimization, at least not in
    the common case.


    Scenario 2: Access following sys_membarrier() must see changes
    from thread that concurrently switches out.

    ----------------------------------------------------------------

    Scheduler sys_membarrier()
    --------- ----------------

    /* begin system call */

    sys_membarrier()
    smp_mb();

    usermode load or store to X

    /* Schedule from user */

    smp_mb();
    rq->curr = new_thread;

    Check rq->curr
    smp_mb();

    smp_mb(); /* not powerpc! */

    /* return to user */

    usermode load or store to Y

    ----------------------------------------------------------------

    Here less ordering is required, given that a read is returning
    the value previously written. Weaker barriers could be used,
    but full memory barriers are in place in any case.


    Potential resolutions, including known stupid ones:

    A. IPI all CPUs all the time. Not so good for real-time workloads,
    and a usermode-induced set of IPIs could potentially be used for
    a denial-of-service (DoS) attack.

    B. Lock all runqueues all the time. This could potentially also be
    used in a usermode-induced DoS attack.

    C. Explicitly interact with all threads rather than with CPUs.
    This can be quite expensive for the surprisingly common case
    where applications have very large numbers of thread. (Java,
    we are looking at you!!!)

    D. Just keep the redundant smp_mb() and just say "no" to Andy's
    x86 optimizations. We would like to avoid the performance
    degradation in both cases.

    E. Require that threads register before using sys_membarrier() for
    private or JIT usage. (The historical implementation using
    synchronize_sched() would continue to -not- require registration,
    both for compatibility and because there is no need to do so.)

    For x86 and powerpc, this registration would set a TIF flag
    on all of the current process's threads. This flag would be
    inherited by any later thread creation within that process, and
    would be cleared by fork() and exec(). When this TIF flag is set,
    the return-to-user path would execute additional code that would
    ensure that ordering and newly JITed code was handled correctly.
    We believe that checks for these TIF flags could be combined with
    existing checks to avoid adding any overhead in the common case
    where the process was not using these sys_membarrier() features.

    For all other architecture, the registration step would be
    a no-op.

    Does anyone have any better solution? If so, please don't keep it
    a secret!

    Thanx, Paul

    \
     
     \ /
      Last update: 2017-09-18 00:36    [W:3.551 / U:0.268 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site