lkml.org 
[lkml]   [2013]   [Jun]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
Date
From
SubjectRe: stop_machine lockup issue in 3.9.y.
On 06/05/2013 12:31 PM, Ben Greear wrote:
> This is no longer really about the module unlink, so changing
> subject.
>
> On 06/05/2013 12:11 PM, Ben Greear wrote:
>> On 06/05/2013 11:48 AM, Tejun Heo wrote:
>>> Hello, Ben.
>>>
>>> On Wed, Jun 05, 2013 at 09:59:00AM -0700, Ben Greear wrote:
>>>> One pattern I notice repeating for at least most of the hangs is that all but one
>>>> CPU thread has irqs disabled and is in state 2. But, there will be one thread
>>>> in state 1 that still has IRQs enabled and it is reported to be in soft-lockup
>>>> instead of hard-lockup. In 'sysrq l' it always shows some IRQ processing,
>>>> but typically that of the sysrq itself. I added printk that would always
>>>> print if the thread notices that smdata->state != curstate, and the soft-lockup
>>>> thread (cpu 2 below) never shows that message.
>>>
>>> It sounds like one of the cpus get live-locked by IRQs. I can't tell
>>> why the situation is made worse by other CPUs being tied up. Do you
>>> ever see CPUs being live locked by IRQs during normal operation?

Hmm, wonder if I found it. I previously saw times where it appears
jiffies does not increment. __do_softirq has a break-out based on
jiffies timeout. Maybe that is failing to get us out of __do_softirq
in my lockup case because for whatever reason the system cannot update
jiffies in this case?

I added this (probably whitespace damaged) hack and now I have not been
able to reproduce the problem.

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 14d7758..621ea3b 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -212,6 +212,7 @@ asmlinkage void __do_softirq(void)
unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
int cpu;
unsigned long old_flags = current->flags;
+ unsigned long loops = 0;

/*
* Mask out PF_MEMALLOC s current task context is borrowed for the
@@ -241,6 +242,7 @@ restart:
unsigned int vec_nr = h - softirq_vec;
int prev_count = preempt_count();

+ loops++;
kstat_incr_softirqs_this_cpu(vec_nr);

trace_softirq_entry(vec_nr);
@@ -265,7 +267,7 @@ restart:

pending = local_softirq_pending();
if (pending) {
- if (time_before(jiffies, end) && !need_resched())
+ if (time_before(jiffies, end) && !need_resched() && (loops < 500))
goto restart;

wakeup_softirqd();
Thanks,
Ben

--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com



\
 
 \ /
  Last update: 2013-06-06 04:21    [W:0.080 / U:0.784 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site