lkml.org 
[lkml]   [2008]   [Apr]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Clock has stopped (time/date looping over 5 seconds), things are broken - what to check to debug?
Joel K. Greene wrote:
> Hi Roger,
>
> Does this sound familiar:
>
> http://lkml.org/lkml/2008/3/14/178

That sounds like it matches what I have.

>
>
> We've been chasing this for quite a while. Our PIC gets in a bad state
> where it thinks the CPU is in the ISR, and so won't give another int. We
> haven't much of an idea of how we get in that state other than that
> HZ=1000 makes it happen faster and HZ=100 causes it less often.

I do have HZ=1000 set, Pavel mentions setting it the =4000 to make it happen
faster, I will try that, I am rebuilding 2.4.24.4 with =4000 in the .config
file, and will verify after it is up that 4000 is running on it.

My machine does have a fair amount of cpu usage (transcoding video), and has a
fair amount of interrupt handling (5 disks, and 3 TV recording cards).

>
> I think that if you look at jiffies you will see it is not incrementing.
> The 4 second loop seems to be in the conversion from jiffies to wall
> time.

I did check the counter in /proc/timer_list under (now at) and it was looping too.

>
>
> It _appears_ that there is a race in the kernel that can be triggered by
> any number of hardware issues. There's another thread by Gregory Stark
> with the same symptoms - he thinks his was fixed by replacing a bad
> DIMM.

I don't think I have bad HW, I will run a test job for a few hours that checks
its results and make sure that the proper answers are coming back, and it is not
crashing.

I do have a couple of disks (on a SIL controller) that every so often appear to
give funny errors, but recover and continue on.

>
> Note that we first saw this on 2.6.16, and Gregory found it on 2.6.5.
> We've seen systems run for a couple of months before seeing this, so
> it's a bear to debug.
>
> How often is this happening for you? How repeatable?

14-30 days, I don't know if it always happens or not, I don't have exact enough
data, but I don't think the machine has made it past 30 days in the last 6
months, if I go back far enough though, I believe it was stable, before I added
a couple of TV recording cards (PVR150, HD5500), and a disk controller (SIL) to
it.

>
> What hardware are you running on?

AMD-754 Sempron64 processor.

ASUS K8V-SE Deluxe MB (VT8385/VT8387 Chipset), so very different HW that the
Serverworks-P3 that you have.









\
 
 \ /
  Last update: 2008-04-07 17:45    [W:0.058 / U:0.524 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site