lkml.org 
[lkml]   [2022]   [Nov]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: PID_NS unshare VS synchronize_rcu_tasks() (was: Re: [Syzkaller & bisect] There is task hung in "synchronize_rcu" in v6.1-rc5 kernel)
Hi Frederic Weisbecker,

On 2022-11-23 at 15:37:58 +0100, Frederic Weisbecker wrote:
> On Mon, Nov 21, 2022 at 01:37:06PM +0800, Pengfei Xu wrote:
> > Hi Frederic Weisbecker and kernel developers,
> >
> > Greeting!
> > There is task hung in "synchronize_rcu" in v6.1-rc5 kernel.
> >
> > Bisected the issue on Raptor and server(No atom small core, big core only),
> > both platforms bisected results show that:
> > first bad commit is c597bfddc9e9e8a63817252b67c3ca0e544ace26:
> > "sched: Provide Kconfig support for default dynamic preempt mode"
> >
> > [ 300.097166] INFO: task rcu_tasks_kthre:11 blocked for more than 147 seconds.
> > [ 300.097455] Not tainted 6.1.0-rc5-094226ad94f4 #1
> > [ 300.097641] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 300.097922] task:rcu_tasks_kthre state:D stack:0 pid:11 ppid:2 flags:0x00004000
> > [ 300.098230] Call Trace:
> > [ 300.098325] <TASK>
> > [ 300.098410] __schedule+0x2de/0x8f0
> > [ 300.098562] schedule+0x5b/0xe0
> > [ 300.098693] schedule_timeout+0x3f1/0x4b0
> > [ 300.098849] ? __sanitizer_cov_trace_pc+0x25/0x60
> > [ 300.099032] ? queue_delayed_work_on+0x82/0xc0
> > [ 300.099206] wait_for_completion+0x81/0x140
> > [ 300.099373] __synchronize_srcu.part.23+0x83/0xb0
> > [ 300.099558] ? __bpf_trace_rcu_stall_warning+0x20/0x20
> > [ 300.099757] synchronize_srcu+0xd6/0x100
> > [ 300.099913] rcu_tasks_postscan+0x19/0x20
> > [ 300.100070] rcu_tasks_wait_gp+0x108/0x290
> > [ 300.100230] ? _raw_spin_unlock+0x1d/0x40
> > [ 300.100389] rcu_tasks_one_gp+0x27f/0x370
> > [ 300.100546] ? rcu_tasks_postscan+0x20/0x20
> > [ 300.100709] rcu_tasks_kthread+0x37/0x50
> > [ 300.100863] kthread+0x14d/0x190
> > [ 300.100998] ? kthread_complete_and_exit+0x40/0x40
> > [ 300.101199] ret_from_fork+0x1f/0x30
> > [ 300.101347] </TASK>
>
> Thanks for reporting this. Fortunately I managed to reproduce and debug.
> It took me a few days to understand the complicated circular dependency
> involved.
>
> So here is a summary:
>
> 1) TASK A calls unshare(CLONE_NEWPID), this creates a new PID namespace
> that every subsequent child of TASK A will belong to. But TASK A doesn't
> itself belong to that new PID namespace.
>
> 2) TASK A forks() and creates TASK B (it is a new threadgroup so it is a
> thread group leader). TASK A stays attached to its PID namespace (let's say PID_NS1)
> and TASK B is the first task belonging to the new PID namespace created by
> unshare() (let's call it PID_NS2).
>
> 3) Since TASK B is the first task attached to PID_NS2, it becomes the PID_NS2
> child reaper.
>
> 4) TASK A forks() again and creates TASK C which get attached to PID_NS2.
> Note how TASK C has TASK A as a parent (belonging to PID_NS1) but has
> TASK B (belonging to PID_NS2) as a pid_namespace child_reaper.
>
> 3) TASK B exits and since it is the child reaper for PID_NS2, it has to
> kill all other tasks attached to PID_NS2, and wait for all of them to die
> before reaping itself (zap_pid_ns_process()). Note it seems to make a
> misleading assumption here, trusting that all tasks in PID_NS2 either
> get reaped by a parent belonging to the same namespace or by TASK B.
> And it is confident that since it deactivated SIGCHLD handler, all
> the remaining tasks ultimately autoreap. And it waits for that to happen.
> However TASK C escapes that rule because it will get reaped by its parent
> TASK A belonging to PID_NS1.
>
> 4) TASK A calls synchronize_rcu_tasks() which leads to
> synchronize_srcu(&tasks_rcu_exit_srcu).
>
> 5) TASK B is waiting for TASK C to get reaped (wrongly assuming it autoreaps)
> But TASK B is under a tasks_rcu_exit_srcu SRCU critical section
> (exit_notify() is between exit_tasks_rcu_start() and
> exit_tasks_rcu_finish()), blocking TASK A
>
> 6) TASK C exits and since TASK A is its parent, it waits for it to reap TASK C,
> but it can't because TASK A waits for TASK B that waits for TASK C.
>
> So there is a circular dependency:
>
> _ TASK A waits for TASK B to get out of tasks_rcu_exit_srcu SRCU critical
> section
> _ TASK B waits for TASK C to get reaped
> _ TASK C waits for TASK A to reap it.
>
> I have no idea how to solve the situation without violating the pid_namespace
> rules and unshare() semantics (although I wish unshare(CLONE_NEWPID) had a less
> error prone behaviour with allowing creating more than one task belonging to the
> same namespace).
>
> So probably having an SRCU read side critical section within exit_notify() is
> not a good idea, is there a solution to work around that for rcu tasks?
>
Thanks for the analysis!
Add one more information: I tried to revert this commit only on top of
v6.1-rc5 mainline by script, but it caused kernel make to fail, it could not
confirm the bisect information is 100% accurate if I could not pass the
revert step verification. I just provide all the information I could.

And this issue is too difficult to me.
If I find more clue, I will update the eamil.

Thanks!
BR.

> Thanks.

\
 
 \ /
  Last update: 2022-11-23 16:46    [W:0.079 / U:1.132 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site