Messages in this thread | | | From | Daniel Bristot de Oliveira <> | Subject | BUG-RT: scheduling while in atomic in the watchdog's hrtimer | Date | Thu, 21 Mar 2019 22:27:30 +0100 |
| |
Hi,
In the 4.19/5.0-rt configured !SMP, the following chain of events can take place:
------------------------------ %< ------------------------------------- smp_apic_timer_interrupt(){ hrtimer_interrupt() { __hrtimer_run_queues() { watchdog_timer_fn() { stop_one_cpu_nowait() { #ifdef !CONFIG_SMP schedule_work() { queue_work() { queue_work_on() { /* phew, long, ah!?.... */ local_lock_irqsave() { __local_lock_irqsave() { __local_lock_irq() { spin_lock_irqsave() { rt_spin_lock()... /* from here on you already know...*/ /* a lot of } */ ------------------------------ >% -------------------------------------
And this can cause a scheduling while in atomic.
Tracking down, the commit that introduced the stop_one_cpu_nowait()-> queue_work_on() to the path was:
------------------------------ %< ------------------------------------- commit 9cf57731b63e37ed995b46690adc604891a9a28f Author: Peter Zijlstra Date: Thu Jun 7 10:52:03 2018 +0200
watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work Oleg suggested to replace the "watchdog/%u" threads with cpu_stop_work. That removes one thread per CPU while at the same time fixes softlockup vs SCHED_DEADLINE. But more importantly, it does away with the single smpboot_update_cpumask_percpu_thread() user, which allows cleanups/shrinkage of the smpboot interface. Suggested-by: Oleg Nesterov Signed-off-by: Peter Zijlstra (Intel) Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: linux-kernel@vger.kernel.org Signed-off-by: Ingo Molnar ------------------------------ >% -------------------------------------
Later modified by:
------------------------------ %< ------------------------------------- commit be45bf5395e0886a93fc816bbe41a008ec2e42e2 Author: Peter Zijlstra Date: Fri Jul 13 12:42:08 2018 +0200
watchdog/softlockup: Fix cpu_stop_queue_work() double-queue bug When scheduling is delayed for longer than the softlockup interrupt period it is possible to double-queue the cpu_stop_work, causing list corruption. Cure this by adding a completion to track the cpu_stop_work's progress. Reported-by: kernel test robot Tested-by: Rong Chen Signed-off-by: Peter Zijlstra (Intel) Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Fixes: 9cf57731b63e ("watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work") Link: http://lkml.kernel.org/r/20180713104208.GW2494@hirez.programming.kicks-ass.net Signed-off-by: Ingo Molnar ------------------------------ >% -------------------------------------
I hit it while trying the automata model against the 4.19-rt/5.0-rt kernels.
You can find more about it here: http://bristot.me/another-bug-found-with-the-model/
Thanks -- Daniel
| |