Messages in this thread |  | | From | Zhouyi Zhou <> | Date | Sun, 1 Jan 2023 21:02:33 +0800 | Subject | Re: [PATCH] torture: Fix hang during kthread shutdown phase |
| |
On Sun, Jan 1, 2023 at 2:16 PM Joel Fernandes (Google) <joel@joelfernandes.org> wrote: > > During shutdown of rcutorture, the shutdown thread in > rcu_torture_cleanup() calls torture_cleanup_begin() which sets fullstop > to FULLSTOP_RMMOD. This is enough to cause the rcutorture threads for > readers and fakewriters to breakout of their main while loop and start > shutting down. > > Once out of their main loop, they then call torture_kthread_stopping() > which in turn waits for kthread_stop() to be called, however > rcu_torture_cleanup() has not even called kthread_stop() on those > threads yet, it does that a bit later. However, before it gets a chance > to do so, torture_kthread_stopping() calls > schedule_timeout_interruptible(1) in a tight loop. Tracing confirmed > this makes the timer softirq constantly execute timer callbacks, while > never returning back to the softirq exit path and is essentially "locked > up" because of that. If the softirq preempts the shutdown thread, > kthread_stop() may never be called. > > This commit improves the situation dramatically, by increasing timeout > passed to schedule_timeout_interruptible() 1/20th of a second. This > causes the timer softirq to not lock up a CPU and everything works fine. > Testing has shown 100 runs of TREE07 passing reliably, which was not the > case before because of RCU stalls. On my Dell PowerEdge R720 with two Intel(R) Xeon(R) CPU E5-2660 128G memory: 1) before this patch: 3 of 80 rounds failed with "rcu: INFO: rcu_sched detected stalls on CPUs/tasks" [1] 2) after this patch all 80 rounds passed
Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Thanks Zhouyi
[1] http://154.220.3.115/logs/20230101/console.log > > Cc: Paul McKenney <paulmck@kernel.org> > Cc: Frederic Weisbecker <fweisbec@gmail.com> > Cc: Zhouyi Zhou <zhouzhouyi@gmail.com> > Cc: <stable@vger.kernel.org> # 6.0.x > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> > --- > kernel/torture.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/kernel/torture.c b/kernel/torture.c > index 29afc62f2bfe..d024f3b7181f 100644 > --- a/kernel/torture.c > +++ b/kernel/torture.c > @@ -915,7 +915,7 @@ void torture_kthread_stopping(char *title) > VERBOSE_TOROUT_STRING(buf); > while (!kthread_should_stop()) { > torture_shutdown_absorb(title); > - schedule_timeout_uninterruptible(1); > + schedule_timeout_uninterruptible(HZ/20); > } > } > EXPORT_SYMBOL_GPL(torture_kthread_stopping); > -- > 2.39.0.314.g84b9a713c41-goog >
|  |