lkml.org 
[lkml]   [2021]   [May]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH v2] rcu: fix a deadlock caused by not release rcu_node->lock
From
Date


On 5/17/21 6:58 AM, Paul E. McKenney wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> On Sun, May 16, 2021 at 05:50:10PM +0800, yanfei.xu@windriver.com wrote:
>> From: Yanfei Xu <yanfei.xu@windriver.com>
>>
>> rcu_node->lock isn't released in rcu_print_task_stall() if the rcu_node
>> don't contain tasks which blocking the GP. However this rcu_node->lock
>> will be used again in rcu_dump_cpu_stacks() soon while the ndetected is
>> non-zero. As a result the cpu will hung by this deadlock.
>>
>> Fixes: c583bcb8f5ed ("rcu: Don't invoke try_invoke_on_locked_down_task() with irqs disabled")
>> Signed-off-by: Yanfei Xu <yanfei.xu@windriver.com>
>
> Also a good catch, thank you! Queued for further review and testing,
> wordsmithed as shown below. The rcutorture scripts have been known to
> work on ARM in the past, and might still do so. (I test on x86.)
>
> As always, please check to make sure that I didn't mess something up.
>

Looks good to me, Thanks!

Regards,
Yanfei

> Thanx, Paul
>
> ------------------------------------------------------------------------
>
> commit e0a9b77f245ae4fe1537120fd5319bf9e091618e
> Author: Yanfei Xu <yanfei.xu@windriver.com>
> Date: Sun May 16 17:50:10 2021 +0800
>
> rcu: Fix stall-warning deadlock due to non-release of rcu_node ->lock
>
> If rcu_print_task_stall() is invoked on an rcu_node structure that does
> not contain any tasks blocking the current grace period, it takes an
> early exit that fails to release that rcu_node structure's lock. This
> results in a self-deadlock, which is detected by lockdep.
>
> To reproduce this bug:
>
> tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 3 --trust-make --configs "TREE03" --kconfig "CONFIG_PROVE_LOCKING=y" --bootargs "rcutorture.stall_cpu=30 rcutorture.stall_cpu_block=1 rcutorture.fwd_progress=0 rcutorture.test_boost=0"
>
> This will also result in other complaints, including RCU's scheduler
> hook complaining about blocking rather than preemption and an rcutorture
> writer stall.
>
> Only a partial RCU CPU stall warning message will be printed because of
> the self-deadlock.
>
> This commit therefore releases the lock on the rcu_print_task_stall()
> function's early exit path.
>
> Fixes: c583bcb8f5ed ("rcu: Don't invoke try_invoke_on_locked_down_task() with irqs disabled")
> Signed-off-by: Yanfei Xu <yanfei.xu@windriver.com>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>
> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> index a10ea1f1f81f..d574e3bbd929 100644
> --- a/kernel/rcu/tree_stall.h
> +++ b/kernel/rcu/tree_stall.h
> @@ -267,8 +267,10 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> struct task_struct *ts[8];
>
> lockdep_assert_irqs_disabled();
> - if (!rcu_preempt_blocked_readers_cgp(rnp))
> + if (!rcu_preempt_blocked_readers_cgp(rnp)) {
> + raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> return 0;
> + }
> pr_err("\tTasks blocked on level-%d rcu_node (CPUs %d-%d):",
> rnp->level, rnp->grplo, rnp->grphi);
> t = list_entry(rnp->gp_tasks->prev,
>

\
 
 \ /
  Last update: 2021-05-17 03:56    [W:0.054 / U:1.876 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site