lkml.org 
[lkml]   [2021]   [Jan]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRE: kdump always hangs in rcu_barrier() -> wait_for_completion()
Date
> From: Paul E. McKenney <paulmck@kernel.org>
> Sent: Thursday, November 26, 2020 3:55 PM
> To: Dexuan Cui <decui@microsoft.com>
> Cc: boqun.feng@gmail.com; Ingo Molnar <mingo@redhat.com>;
> rcu@vger.kernel.org; vkuznets <vkuznets@redhat.com>; Michael Kelley
> <mikelley@microsoft.com>; linux-kernel@vger.kernel.org
> Subject: Re: kdump always hangs in rcu_barrier() -> wait_for_completion()
>
> On Thu, Nov 26, 2020 at 10:59:19PM +0000, Dexuan Cui wrote:
> > > From: Paul E. McKenney <paulmck@kernel.org>
> > > Sent: Thursday, November 26, 2020 1:42 PM
> > >
> > > > > Another possibility is that rcu_state.gp_kthread is non-NULL, but that
> > > > > something else is preventing RCU grace periods from completing, but in
> > > >
> > > > It looks like somehow the scheduling is not working here: in rcu_barrier()
> > > > , if I replace the wait_for_completion() with
> > > > wait_for_completion_timeout(&rcu_state.barrier_completion, 30*HZ),
> the
> > > > issue persists.
> > >
> > > Have you tried using sysreq-t to see what the various tasks are doing?
> >
> > Will try it.
> >
> > BTW, this is a "Generation 2" VM on Hyper-V, meaning sysrq only starts to
> > work after the Hyper-V para-virtualized keyboard driver loads... So, at this
> > early point, sysrq is not working. :-( I'll have to hack the code and use a
> > virtual NMI interrupt to force the sysrq handler to be called.
>
> Whatever works!
>
> > > Having interrupts disabled on all CPUs would have the effect of disabling
> > > the RCU CPU stall warnings.
> > > Thanx, Paul
> >
> > I'm sure the interrupts are not disabled. Here the VM only has 1 virtual CPU,
> > and when the hang issue happens the virtual serial console is still responding
> > when I press Enter (it prints a new line) or Ctrl+C (it prints ^C).
> >
> > Here the VM does not use the "legacy timers" (PIT, Local APIC timer, etc.) at
> all.
> > Instead, the VM uses the Hyper-V para-virtualized timers. It looks the
> Hyper-V
> > timer never fires in the kdump kernel when the hang issue happens. I'm
> > looking into this... I suspect this hang issue may only be specific to Hyper-V.
>
> Fair enough, given that timers not working can also suppress RCU CPU
> stall warnings. ;-)
>
> Thanx, Paul

FYI: the issue has been fixed by this fix:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fff7b5e6ee63c5d20406a131b260c619cdd24fd1

Thanks,
-- Dexuan

\
 
 \ /
  Last update: 2021-01-28 08:32    [W:0.141 / U:0.348 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site