lkml.org 
[lkml]   [2021]   [Oct]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH] rcu/nocb: Fix misordered rcu_barrier() while (de-)offloading
On Mon, Oct 18, 2021 at 07:42:42PM +0200, Frederic Weisbecker wrote:
> On Mon, Oct 18, 2021 at 09:18:14AM -0700, Paul E. McKenney wrote:
> > On Mon, Oct 18, 2021 at 01:32:59PM +0200, Frederic Weisbecker wrote:
> > > When an rdp is in the process of (de-)offloading, rcu_core() and the
> > > nocb kthreads can process callbacks at the same time. This leaves many
> > > possible scenarios leading to an rcu barrier to execute before
> > > the preceding callbacks. Here is one such example:
> > >
> > > CPU 0 CPU 1
> > > -------------- ---------------
> > > call_rcu(callbacks1)
> > > call_rcu(callbacks2)
> > > // move callbacks1 and callbacks2 on the done list
> > > rcu_advance_callbacks()
> > > call_rcu(callbacks3)
> > > rcu_barrier_func()
> > > rcu_segcblist_entrain(...)
> > > nocb_cb_wait()
> > > rcu_do_batch()
> > > callbacks1()
> > > cond_resched_tasks_rcu_qs()
> > > // move callbacks3 and rcu_barrier_callback()
> > > // on the done list
> > > rcu_advance_callbacks()
> > > rcu_core()
> > > rcu_do_batch()
> > > callbacks3()
> > > rcu_barrier_callback()
> > > //MISORDERING
> > > callbacks2()
> > >
> > > Fix this with preventing two concurrent rcu_do_batch() on a same rdp
> > > as long as an rcu barrier callback is pending somewhere.
> > >
> > > Reported-by: Paul E. McKenney <paulmck@kernel.org>
> > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > Cc: Josh Triplett <josh@joshtriplett.org>
> > > Cc: Joel Fernandes <joel@joelfernandes.org>
> > > Cc: Boqun Feng <boqun.feng@gmail.com>
> > > Cc: Neeraj Upadhyay <neeraju@codeaurora.org>
> > > Cc: Uladzislau Rezki <urezki@gmail.com>
> >
> > Yow!
> >
> > But how does the (de-)offloading procedure's acquisition of
> > rcu_state.barrier_mutex play into this? In theory, that mutex was
> > supposed to prevent these sorts of scenarios. In practice, it sounds
> > like the shortcomings in this theory should be fully explained so that
> > we don't get similar bugs in the future. ;-)
>
> I think you're right. The real issue is something I wanted to
> fix next: RCU_SEGCBLIST_RCU_CORE isn't cleared when nocb is enabled on
> boot so rcu_core() always run concurrently with nocb kthreads in TREE04,
> without holding rcu_barrier mutex of course (I mean with the latest patchset).

That would do it!

> Ok forget this patch, I'm testing again with simply clearing
> RCU_SEGCBLIST_RCU_CORE on boot.

Sounds good, looking forward to it!

Thanx, Paul

\
 
 \ /
  Last update: 2021-10-18 20:37    [W:0.192 / U:0.324 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site