Messages in this thread Patch in this message | | | From | Frederic Weisbecker <> | Subject | [PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly | Date | Wed, 16 Mar 2022 15:42:54 +0100 |
| |
The workqueue ordering guarantees that the work sees all the accesses of the task prior to its call to the corresponding queue_work().
Therefore the sequence to poll can be retrieved locklessly.
The only downside is that it is then possible to miss the 0x1 flag set by a prior work. But this could already happen concurrently anyway after the exp_poll_lock is unlocked. In the worst case the slow path involving synchronize_rcu_expedited() takes care of the situation.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org> Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Uladzislau Rezki <uladzislau.rezki@sony.com> Cc: Joel Fernandes <joel@joelfernandes.org> --- kernel/rcu/tree_exp.h | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 763ec35546ed..c4a19c6a83cf 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -909,9 +909,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp) struct rcu_node *rnp = container_of(wp, struct rcu_node, exp_poll_wq); unsigned long s; - raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); - s = rnp->exp_seq_poll_rq; - raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); + s = READ_ONCE(rnp->exp_seq_poll_rq); if (s & 0x1) return; while (!sync_exp_work_done(s)) @@ -919,7 +917,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); s = rnp->exp_seq_poll_rq; if (!(s & 0x1) && sync_exp_work_done(s)) - rnp->exp_seq_poll_rq |= 0x1; + WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1); raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); } @@ -949,7 +947,7 @@ unsigned long start_poll_synchronize_rcu_expedited(void) if (rcu_init_invoked()) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)) { - rnp->exp_seq_poll_rq = s; + WRITE_ONCE(rnp->exp_seq_poll_rq, s); if (rcu_init_invoked()) queue_work(rcu_gp_wq, &rnp->exp_poll_wq); } -- 2.25.1
| |