lkml.org 
[lkml]   [2021]   [Oct]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 1/2] powerpc/watchdog: prevent printk and send IPI while holding the wd lock
    Excerpts from Laurent Dufour's message of October 27, 2021 6:14 pm:
    > Le 27/10/2021 à 05:29, Nicholas Piggin a écrit :
    >> Excerpts from Laurent Dufour's message of October 27, 2021 2:27 am:
    >>> When handling the Watchdog interrupt, long processing should not be done
    >>> while holding the __wd_smp_lock. This prevents the other CPUs to grab it
    >>> and to process Watchdog timer interrupts. Furhtermore, this could lead to
    >>> the following situation:
    >>>
    >>> CPU x detect lockup on CPU y and grab the __wd_smp_lock
    >>> in watchdog_smp_panic()
    >>> CPU y caught the watchdog interrupt and try to grab the __wd_smp_lock
    >>> in soft_nmi_interrupt()
    >>> CPU x wait for CPU y to catch the IPI for 1s in __smp_send_nmi_ipi()
    >>
    >> CPU y should get the IPI here if it's a NMI IPI (which will be true for
    >>> = POWER9 64s).
    >>
    >> That said, not all platforms support it and the console lock problem
    >> seems real, so okay.
    >>
    >>> CPU x will timeout and so has spent 1s waiting while holding the
    >>> __wd_smp_lock.
    >>>
    >>> A deadlock may also happen between the __wd_smp_lock and the console_owner
    >>> 'lock' this way:
    >>> CPU x grab the console_owner
    >>> CPU y grab the __wd_smp_lock
    >>> CPU x catch the watchdog timer interrupt and needs to grab __wd_smp_lock
    >>> CPU y wants to print something and wait for console_owner
    >>> -> deadlock
    >>>
    >>> Doing all the long processing without holding the _wd_smp_lock prevents
    >>> these situations.
    >>
    >> The intention was to avoid logs getting garbled e.g., if multiple
    >> different CPUs fire at once.
    >>
    >> I wonder if instead we could deal with that by protecting the IPI
    >> sending and printing stuff with a trylock, and if you don't get the
    >> trylock then just return, and you'll come back with the next timer
    >> interrupt.
    >
    > That sounds a bit risky to me, especially on large system when system goes
    > wrong, all the CPU may try lock here.

    That should be okay though, one will get through and the others will
    skip.

    > Furthermore, now operation done under the lock protection are quite fast, there
    > is no more spinning like the delay loop done when sending an IPI.
    >
    > Protecting the IPI sending is a nightmare, if the target CPU is later play with
    > the lock we are taking during the IPI processing, furthermore, if the target CPU
    > is not responding the sending CPU is waiting for 1s, which slows all the system
    > due to the lock held.
    > Since I do a copy of the pending CPU mask and clear it under the lock
    > protection, the IPI sending is safe while done without holding the lock.

    Protecting IPI sending basically has all the same issues in the NMI
    IPI layer.

    >
    > Regarding the interleaved traces, I don't think this has to be managed down
    > here, but rather in the printk/console path.

    It can't necessarily be because some of the problem is actually that a
    NMI handler can be interrupted by another NMI IPI because the caller
    can return only after handlers start running rather than complete.

    I don't think it would be an additional nightmare to trylock.

    Thanks,
    Nick

    \
     
     \ /
      Last update: 2021-10-27 10:51    [W:2.388 / U:0.108 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site