Messages in this thread | | | Date | Thu, 21 May 2020 22:00:33 -0400 | From | Qian Cai <> | Subject | Re: Endless soft-lockups for compiling workload since next-20200519 |
| |
On Thu, May 21, 2020 at 11:39:38AM +0200, Peter Zijlstra wrote: > On Thu, May 21, 2020 at 02:40:36AM +0200, Frederic Weisbecker wrote: > > On Wed, May 20, 2020 at 02:50:56PM +0200, Peter Zijlstra wrote: > > > On Tue, May 19, 2020 at 11:58:17PM -0400, Qian Cai wrote: > > > > Just a head up. Repeatedly compiling kernels for a while would trigger > > > > endless soft-lockups since next-20200519 on both x86_64 and powerpc. > > > > .config are in, > > > > > > Could be 90b5363acd47 ("sched: Clean up scheduler_ipi()"), although I've > > > not seen anything like that myself. Let me go have a look. > > > > > > > > > In as far as the logs are readable (they're a wrapped mess, please don't > > > do that!), they contain very little useful, as is typical with IPIs :/ > > > > > > > [ 1167.993773][ C1] WARNING: CPU: 1 PID: 0 at kernel/smp.c:127 > > > > flush_smp_call_function_queue+0x1fa/0x2e0 > > > > So I've tried to think of a race that could produce that and here is > > the only thing I could come up with. It's a bit complicated unfortunately: > > This: > > > smp_call_function_single_async() { smp_call_function_single_async() { > > // verified csd->flags != CSD_LOCK // verified csd->flags != CSD_LOCK > > csd->flags = CSD_LOCK csd->flags = CSD_LOCK > > concurrent smp_call_function_single_async() using the same csd is what > I'm looking at as well. Now in the ILB case there is an easy cure: > > (because there is only a single ilb target) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 01f94cf52783..b6d8a7b991f0 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -10033,7 +10033,7 @@ static void kick_ilb(unsigned int flags) > * is idle. And the softirq performing nohz idle load balance > * will be run before returning from the IPI. > */ > - smp_call_function_single_async(ilb_cpu, &cpu_rq(ilb_cpu)->nohz_csd); > + smp_call_function_single_async(ilb_cpu, &this_rq()->nohz_csd); > } > > /* > > Qian, can you give that a spin?
Running for a few hours now. It works fine.
| |