Messages in this thread | | | From | Sergey Senozhatsky <> | Subject | [RFC][PATCH 0/4] printk: introduce printing kernel thread | Date | Mon, 6 Mar 2017 21:45:50 +0900 |
| |
Hello,
RFC
This patch set adds a printk() kernel thread which lets us to print kernel messages to the console from a non-atomic/schedule-able context, avoiding different sort of lockups, stalls, etc.
The patch set is in [RFC] stage - it may contain ugly and dumb ideas. The reason I decided to take a step back is because we've been using printk kthread in development/production for ~10 months now and there are some observations from my side, which I want to list/outline.
1) We had a `bool' flag to temporarily disable printk offloading (in suspend, hibernate, etc.) to address a bug reported by Viresh Kumar [1]. Turned out that our in-house kernel tweaks/ad hocs in PM code introduce nested sections that disable/enable printk offloading and thus `bool' flag can re-enable printk-kthread too early. (it wasn't me who added those PM 'tweaks' :)).
Would there be any objections if we switch to `int' flag and allow nested printk-kthread enable/disable sections?
2) Just letting you know, printk kthread changes the behavior of printk in one _corner case_. The corner case is quite interesting and actually consists of two corner cases. Suppose on SMP system there is only one CPU that printk()-s a lot, the rest of CPUs don't lock console_sem and don't printk(). Previously that printing CPU had been throttling itself (*) because of console drivers call for every printk():
CPU0
printk("a") console_unlock() call_console_drivers("a")
...
printk("z") console_unlock() call_console_drivers("z")
* Given that no other CPU locks the console_sem.
With printk kthread the case turns into this one:
CPU0 CPU1
printk("a") wake_up printk_kthread ... printk_kthread printk("k") console_unlock() ... call_console_drivers("a") printk("z") call_console_drivers("b") call_console_drivers("c") ...
The second 'corner case' part here is that CPU0 may be much faster than printing CPU, which may result in dropped printk messages.
This all is absolutely possible even with out the printk-kthread. A single console_lock() call from CPUx will result in exactly the same condition. So it's not necessarily a regression. But there may be some scenarios in the kernel that may suffer from this change. From the top of my head -- sysrq backtrace dump, and, probably, OOM print out and backtrace dump.
sysrq is potentially even trickier. can we always wake_up() kernel thread from sysrq? there probably might be cases when we can't rely on the scheduler.
So may be we would want to export disable/enable printk-kthread offloading symbols and let people use them in appropriate places.
What do you think?
3) A minor one, 'async printk' is, apparently, a confusing name. people assume that the opposite to 'asynchronous printk' is 'synchronous printk'. which is not the case. printk may accidentally be synchronous but in general there is no way to make it always synchronous on SMP system for all CPUs (well, except for wrapping every printk() in console_lock()/console_unlock(), probably), because it's up to the CPU that holds the console_sem to print kernel messages from console_unlock() at some point. so in some sense it's asynchronous by design. printk-kthread targets different aims anyway. but still I would probably prefer to avoid this confusion.
[1] this is from the days when printk-kthread had RT prio lkml.kernel.org/r/20160714221251.GE3057@ubuntu
= Patches 0001/0002 just introduce the printk kthread, it's not eanbled yet.
= Patch 0003 is incomplete and a bit ugly. I need to think more. It exposes some printk internals and in general I'm not really happy with it. PM parts, probably, can be kept within the printk.c. For example, disable printk-kthread in suspend_console() and enable it in resume_console(). unlike sysrq though.
= Patch 0004 enables printk-kthread.
Sergey Senozhatsky (4): printk: introduce printing kernel thread printk: offload printing from wake_up_klogd_work_func() kernel, power: disable printk_kthread in unsafe places printk: enable printk offloading
include/linux/console.h | 3 ++ kernel/kexec_core.c | 4 ++ kernel/power/hibernate.c | 8 +++ kernel/power/suspend.c | 4 ++ kernel/printk/printk.c | 133 ++++++++++++++++++++++++++++++++++++++++++++--- 5 files changed, 144 insertions(+), 8 deletions(-)
-- 2.12.0
| |