Messages in this thread Patch in this message | | | From | Zhenzhong Duan <> | Subject | [PATCH 1/2] acpi_pm: Fix bootup softlockup due to PMTMR counter read contention | Date | Thu, 14 Mar 2019 16:42:11 +0800 |
| |
During bootup stage of a large system with many CPUs, with nohpet, PMTMR is temporarily selected as the clock source which can lead to a softlockup because of the following reasons: 1) There is a single PMTMR counter shared by all the CPUs. 2) PMTMR counter reading is a very slow operation.
At bootup stage tick device is firstly initialized in periodic mode and then switch to one-shot mode when a high resolution clocksource is initialized. Between clocksoure initialization and switching to one-shot mode, there is small window where timer interrupt triggers.
Due to PMTMR read contention, the 1ms(HZ=1000) interval isn't enough for all the CPUs to process timer interrupt in periodic mode. Then CPUs are busy processing interrupt one by one without a break, tick_clock_notify() have no chance to be called and we never switch to one-shot mode. Finally the system may crash because of a NMI watchdog soft lockup, logs:
[ 20.181521] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 44.273786] BUG: soft lockup - CPU#48 stuck for 23s! [swapper/48:0] [ 44.279992] BUG: soft lockup - CPU#49 stuck for 23s! [migration/49:307] [ 44.285169] BUG: soft lockup - CPU#50 stuck for 23s! [migration/50:313]
In one-shot mode, the contention is still there but next event is always set with a future value. We may missed some ticks, but the timer code is smart enough to pick up those missed ticks.
By moving tick_clock_notify() into stop_machine, kernel changes to one-shot mode early before the contention accumulate and lockup system.
This patch also address the same issue of commit f99fd22e4d4b ("x86/hpet: Reduce HPET counter read contention") in a simple way, so that commit could be reverted.
Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com> Tested-by: Kin Cho <kin.cho@oracle.com> Cc: John Stultz <john.stultz@linaro.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Stephen Boyd <sboyd@kernel.org> Cc: Waiman Long <longman@redhat.com> Cc: Srinivas Eeda <srinivas.eeda@oracle.com> --- kernel/time/timekeeping.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index f986e19..815c92d 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -1378,6 +1378,7 @@ static int change_clocksource(void *data) write_seqcount_end(&tk_core.seq); raw_spin_unlock_irqrestore(&timekeeper_lock, flags); + tick_clock_notify(); return 0; } @@ -1396,7 +1397,6 @@ int timekeeping_notify(struct clocksource *clock) if (tk->tkr_mono.clock == clock) return 0; stop_machine(change_clocksource, clock, NULL); - tick_clock_notify(); return tk->tkr_mono.clock == clock ? 0 : -1; } -- 1.8.3.1
| |