lkml.org 
[lkml]   [2015]   [Sep]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 0/4] printk: Softlockup avoidance
    On Wed, 19 Aug 2015 17:38:27 +0200 Jan Kara <jack@suse.com> wrote:

    > From: Jan Kara <jack@suse.cz>
    >
    > Hello,
    >
    > since lately there were several attempts at dealing with softlockups due
    > to heavy printk traffic [1] [2] and I've been also privately pinged by
    > couple of people about the state of the patch set, I've decided to respin
    > the patch set.
    >
    > To remind the original problem:
    >
    > Currently, console_unlock() prints messages from kernel printk buffer to
    > console while the buffer is non-empty. When serial console is attached,
    > printing is slow and thus other CPUs in the system have plenty of time
    > to append new messages to the buffer while one CPU is printing. Thus the
    > CPU can spend unbounded amount of time doing printing in console_unlock().
    > This is especially serious when printk() gets called under some critical
    > spinlock or with interrupts disabled.
    >
    > In practice users have observed a CPU can spend tens of seconds printing
    > in console_unlock() (usually during boot when hundreds of SCSI devices
    > are discovered) resulting in RCU stalls (CPU doing printing doesn't
    > reach quiescent state for a long time), softlockup reports (IPIs for the
    > printing CPU don't get served and thus other CPUs are spinning waiting
    > for the printing CPU to process IPIs), and eventually a machine death
    > (as messages from stalls and lockups append to printk buffer faster than
    > we are able to print). So these machines are unable to boot with serial
    > console attached. Also during artificial stress testing SATA disk
    > disappears from the system because its interrupts aren't served for too
    > long.
    >
    > This series addresses the problem in the following way: If CPU has printed
    > more that printk_offload (defaults to 1000) characters, it wakes up one
    > of dedicated printk kthreads (we don't use workqueue because that has
    > deadlock potential if printk was called from workqueue code). Once we find
    > out kthread is spinning on a lock, we stop printing, drop console_sem, and
    > let kthread continue printing. Since there are two printing kthreads, they
    > will pass printing between them and thus no CPU gets hogged by printing.

    I still hate your patchset ;)

    But nothing better suggests itself. I have a few review comments -
    please let's work through that stuff, get a fresh version out and we'll
    see how it goes.

    Is this patchset being used in the field? Perhaps in the suse kernel?
    If so, a mention of that in the changelog would help things along.



    \
     
     \ /
      Last update: 2015-09-19 00:41    [W:5.021 / U:0.176 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site