lkml.org 
[lkml]   [2013]   [Jul]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Yet more softlockups.
    On 07/12/2013 08:45 AM, Dave Jones wrote:
    > On Fri, Jul 12, 2013 at 08:38:52AM -0700, Dave Hansen wrote:
    > > Dave, for your case, my suspicion would be that it got turned on
    > > inadvertently, or that we somehow have a bug which bumped up
    > > perf_event.c's 'active_events' and we're running some perf code that we
    > > don't have to.
    >
    > What do you 'inadvertantly' ? I see this during bootup every time.
    > Unless systemd or something has started playing with perf, (which afaik it isn't)

    I mean that somebody turned 'active_events' on without actually wanting
    perf to be on. I'd be curious how it got set to something nonzero.
    Could you stick a WARN_ONCE() or printk_ratelimit() on the three sites
    that modify it?

    > > But, I'm suspicious. I was having all kinds of issues with perf and
    > > NMIs taking hundreds of milliseconds. I never isolated it to having a
    > > real, single, cause. I attributed it to my large NUMA system just being
    > > slow. Your description makes me wonder what I missed, though.
    >
    > Here's a fun trick:
    >
    > trinity -c perf_event_open -C4 -q -l off
    >
    > Within about a minute, that brings any of my boxes to its knees.
    > The softlockup detector starts going nuts, and then the box wedges solid.

    On my box, the same happens with 'perf top'. ;)

    *But* dropping the perf sample rate has been really effective at keeping
    me from hitting it, and I've had to use _lots_ of CPUs (60-160) doing
    those NMIs at once to trigger the lockups.

    Being able to trigger it with so few CPUs is interesting though. I'll
    try on my hardware.


    \
     
     \ /
      Last update: 2013-07-12 18:21    [W:7.206 / U:0.208 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site