Messages in this thread | | | Date | Mon, 25 Apr 2022 17:03:28 -0700 | From | "Paul E. McKenney" <> | Subject | Re: "Dying CPU not properly vacated" splat |
| |
On Mon, Apr 25, 2022 at 10:59:44PM +0100, Valentin Schneider wrote: > On 25/04/22 10:33, Paul E. McKenney wrote: > > On Mon, Apr 25, 2022 at 05:15:13PM +0100, Valentin Schneider wrote: > >> > >> Hi Paul, > >> > >> On 21/04/22 12:38, Paul E. McKenney wrote: > >> > Hello! > >> > > >> > The rcutorture TREE03 scenario got the following splat, which appears > >> > to be a one-off, or if not, having an MTBF in the thousands of hours, > >> > even assuming that it is specific to TREE03. (If it is not specific to > >> > TREE03, we are talking tens of thousands of hours of rcutorture runtime.) > >> > > >> > So just in case this rings any bells or there are some diagnostics I > >> > should add in case this ever happens again. ;-) > >> > >> There should be a dump of the enqueued tasks right after the snippet you've > >> sent, any chance you could share that if it's there? That should tell us > >> which tasks are potentially misbehaving. > > > > And now that I know to look for them, there they are! Thank you!!! > > > > CPU7 enqueued tasks (2 total): > > pid: 52, name: migration/7 > > pid: 135, name: rcu_torture_rea > > smpboot: CPU 7 is now offline > > > > So what did rcu_torture_reader() do wrong here? ;-) > > > > So on teardown, CPUHP_AP_SCHED_WAIT_EMPTY->sched_cpu_wait_empty() waits for > the rq to be empty. Tasks must *not* be enqueued onto that CPU after that > step has been run - if there are per-CPU tasks bound to that CPU, they must > be unbound in their respective hotplug callback. > > For instance for workqueue.c, we have workqueue_offline_cpu() as a hotplug > callback which invokes unbind_workers(cpu), the interesting bit being: > > for_each_pool_worker(worker, pool) { > kthread_set_per_cpu(worker->task, -1); > WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0); > } > > The rcu_torture_reader() kthreads aren't bound to any particular CPU are > they? I can't find any code that would indicate they are - and in that case > it means we have a problem with is_cpu_allowed() or related.
I did not intend that the rcu_torture_reader() kthreads be bound, and I am not seeing anything that binds them.
Thoughts? (Other than that validating any alleged fix will be quite "interesting".)
Thanx, Paul
| |