Messages in this thread | | | From | Joel Fernandes <> | Date | Thu, 18 Aug 2022 19:05:19 -0400 | Subject | Re: [PATCH v3 resend 4/6] fs: Move call_rcu() to call_rcu_lazy() in some paths |
| |
On Thu, Aug 18, 2022 at 1:23 PM Joel Fernandes <joel@joelfernandes.org> wrote: > > [Sorry, adding back the CC list] > > On Mon, Aug 8, 2022 at 11:45 PM Joel Fernandes (Google) > <joel@joelfernandes.org> wrote: > > > > This is required to prevent callbacks triggering RCU machinery too > > quickly and too often, which adds more power to the system. > > > > When testing, we found that these paths were invoked often when the > > system is not doing anything (screen is ON but otherwise idle). > > Unfortunately, I am seeing a slow down in ChromeOS boot performance > after applying this particular patch. It is the first time I could > test ChromeOS boot times with the series since it was hard to find a > ChromeOS device that runs the upstream kernel. > > Anyway, Vlad, Neeraj, do you guys also see slower boot times with this > patch? I wonder if the issue is with wake up interaction with the nocb > GP threads. > > We ought to disable lazy RCU during boot since it would have little > benefit anyway. But I am also concerned about some deeper problem I > did not catch before. > > I'll look into tracing the fs paths to see if I can narrow down what's > causing it. Will also try a newer kernel, I am currently testing on > 5.19-rc4.
I got somewhere with this. It looks like queuing CBs as lazy CBs instead of normal CBs, are triggering expedited stalls during the boot process:
39.949198] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { } 28 jiffies s: 69 root: 0x0/.
No idea how/why lazy RCU CBs would be related to expedited GP issues, but maybe something hangs and causes that side-effect.
initcall_debug did not help, as it seems initcalls all work fine, and then 8 seconds after the boot, it starts slowing down a lot, followed by the RCU stall messages. As a next step I'll enable ftrace during the boot to see if I can get more insight. But I believe, its not the FS layer, the FS layer just triggers lazy CBs, but there is something wrong with the core lazy-RCU work itself.
This kernel is 5.19-rc4. I'll also try to rebase ChromeOS on more recent kernels and debug.
Thanks,
- Joel
| |