lkml.org 
[lkml]   [2022]   [Mar]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
From
Subject[PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly
Date
The workqueue ordering guarantees that the work sees all the accesses
of the task prior to its call to the corresponding queue_work().

Therefore the sequence to poll can be retrieved locklessly.

The only downside is that it is then possible to miss the 0x1 flag set
by a prior work. But this could already happen concurrently anyway after
the exp_poll_lock is unlocked. In the worst case the slow path involving
synchronize_rcu_expedited() takes care of the situation.

Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Uladzislau Rezki <uladzislau.rezki@sony.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/tree_exp.h | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 763ec35546ed..c4a19c6a83cf 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -909,9 +909,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
struct rcu_node *rnp = container_of(wp, struct rcu_node, exp_poll_wq);
unsigned long s;

- raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
- s = rnp->exp_seq_poll_rq;
- raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
+ s = READ_ONCE(rnp->exp_seq_poll_rq);
if (s & 0x1)
return;
while (!sync_exp_work_done(s))
@@ -919,7 +917,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
s = rnp->exp_seq_poll_rq;
if (!(s & 0x1) && sync_exp_work_done(s))
- rnp->exp_seq_poll_rq |= 0x1;
+ WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1);
raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
}

@@ -949,7 +947,7 @@ unsigned long start_poll_synchronize_rcu_expedited(void)
if (rcu_init_invoked())
raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)) {
- rnp->exp_seq_poll_rq = s;
+ WRITE_ONCE(rnp->exp_seq_poll_rq, s);
if (rcu_init_invoked())
queue_work(rcu_gp_wq, &rnp->exp_poll_wq);
}
--
2.25.1
\
 
 \ /
  Last update: 2022-03-16 15:45    [W:0.137 / U:1.368 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site