Messages in this thread | | | From | Petr Mladek <> | Subject | [RFC 0/5] workqueue: Debugging improvements | Date | Wed, 1 Feb 2023 14:45:38 +0100 |
| |
The workqueue watchdog provides a lot of impormation when a stall is detected. The report says a lot about what workqueues and worker pools are active and what is being blocked. Unfortunately, it does not provide much information about what caused the stall.
In particular, it did not help me to get root of the following problems:
+ New workers were not created because the system reached PID limit. Admins limited it too much in a cloud.
+ A networking driver was not loaded because systemd killed modprobe when switching the root from initrd to the booted system.
It was surprisingly quite reproducible. Interrupts are not handled immediately in kernel code. The wait in kthread_create_on_node() was one of few locations. So the race window evidently was not trivial.
1st patch fixes a misleading "hung" time report.
2nd, 3rd, and 4rd patches add warnings into create_worker() and create_rescuer(). The rather persistent errors are printed only once until it succeeds again. Otherwise it would be too noisy and might even break the watchdog.
5th patch adds printing bracktraces of CPU-bound workers that might block CPU-bound workqueues. The candidate is well defined to keep the number of backtraces small. It always printed only the right one during my testing.
The first 4 patches would have helped me to debug the real problems that I met.
The 5th patch is theoretical. I did not see this case in practice. But it looks realistic enough. And it worked very well when I simulated the problem. IMHO, it should be pretty useful.
Petr Mladek (5): workqueue: Fix hung time report of worker pools workqueue: Warn when a new worker could not be created workqueue: Interrupted create_worker() is not a repeated event workqueue: Warn when a rescuer could not be created workqueue: Print backtraces from CPUs with hung CPU bound workqueues
kernel/workqueue.c | 186 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 177 insertions(+), 9 deletions(-)
-- 2.35.3
| |