Messages in this thread | | | From | Stephen Brennan <> | Subject | [PATCH v2 0/4] printk: reduce deadlocks during panic | Date | Wed, 26 Jan 2022 15:02:32 -0800 |
| |
When a caller writes heavily to the kernel log (e.g. writing to /dev/kmsg in a loop) while another panics, there's currently a high likelihood of a deadlock (see patch 2 for the full description of this deadlock).
The principle fix is to disable the optimistic spin once panic_cpu is set, so the panic CPU doesn't spin waiting for a halted CPU to hand over the console_sem.
However, this exposed us to a livelock situation, where the panic CPU holds the console_sem, and another CPU could fill up the log buffer faster than the consoles could drain it, preventing the panic from progressing and halting the other CPUs. To avoid this, patch 3 adds a mechanism to suppress printk (from non-panic-CPU) during panic, if we reach a threshold of dropped messages.
A major goal with all of these patches is to try to decrease the likelihood that another CPU is holding the console_sem when we halt it in panic(). This reduces the odds of needing to break locks and potentially encountering further deadlocks with the console drivers.
To test, I use the following script, kmsg_panic.sh:
#!/bin/bash date # 991 chars (based on log buffer size): chars="$(printf 'a%.0s' {1..991})" while :; do echo $chars > /dev/kmsg done & echo c > /proc/sysrq-trigger & date exit
I defined a hang as any time the system did not reboot to a login prompt on the serial console within 60 seconds. Here are the statistics on hangs using this script, before and after the patch.
before: 776 hangs / 1484 trials - 52.3% after : 0 hangs / 408 trials - 0.0%
v2: Each patch has minor updates from code reviews. I've re-done testing and updated the above statistics. Exact changes are in each patch.
Stephen Brennan (4): panic: Add panic_in_progress helper printk: disable optimistic spin during panic printk: Avoid livelock with heavy printk during panic printk: Drop console_sem during panic
include/linux/panic.h | 6 +++++ kernel/printk/printk.c | 51 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)
-- 2.30.2
| |