Messages in this thread | | | From | Valentin Schneider <> | Subject | Re: "Dying CPU not properly vacated" splat | Date | Mon, 25 Apr 2022 22:59:44 +0100 |
| |
On 25/04/22 10:33, Paul E. McKenney wrote: > On Mon, Apr 25, 2022 at 05:15:13PM +0100, Valentin Schneider wrote: >> >> Hi Paul, >> >> On 21/04/22 12:38, Paul E. McKenney wrote: >> > Hello! >> > >> > The rcutorture TREE03 scenario got the following splat, which appears >> > to be a one-off, or if not, having an MTBF in the thousands of hours, >> > even assuming that it is specific to TREE03. (If it is not specific to >> > TREE03, we are talking tens of thousands of hours of rcutorture runtime.) >> > >> > So just in case this rings any bells or there are some diagnostics I >> > should add in case this ever happens again. ;-) >> >> There should be a dump of the enqueued tasks right after the snippet you've >> sent, any chance you could share that if it's there? That should tell us >> which tasks are potentially misbehaving. > > And now that I know to look for them, there they are! Thank you!!! > > CPU7 enqueued tasks (2 total): > pid: 52, name: migration/7 > pid: 135, name: rcu_torture_rea > smpboot: CPU 7 is now offline > > So what did rcu_torture_reader() do wrong here? ;-) >
So on teardown, CPUHP_AP_SCHED_WAIT_EMPTY->sched_cpu_wait_empty() waits for the rq to be empty. Tasks must *not* be enqueued onto that CPU after that step has been run - if there are per-CPU tasks bound to that CPU, they must be unbound in their respective hotplug callback.
For instance for workqueue.c, we have workqueue_offline_cpu() as a hotplug callback which invokes unbind_workers(cpu), the interesting bit being:
for_each_pool_worker(worker, pool) { kthread_set_per_cpu(worker->task, -1); WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0); }
The rcu_torture_reader() kthreads aren't bound to any particular CPU are they? I can't find any code that would indicate they are - and in that case it means we have a problem with is_cpu_allowed() or related.
| |