Messages in this thread | | | Date | Wed, 30 Nov 2022 09:38:00 +0800 | From | Feng Tang <> | Subject | Re: [PATCH clocksource 1/3] clocksource: Reject bogus watchdog clocksource measurements |
| |
On Tue, Nov 29, 2022 at 11:29:15AM -0800, Paul E. McKenney wrote: [...] > > > > IIUC, this will make TSC to watchdog HPET every 500 ms. We have got > > > > report that the 500ms watchdog timer had big impact on some parallel > > > > workload on big servers, that was another factor for us to seek > > > > stopping the timer. > > > > > > Another approach would be to slow it down. Given the tighter bounds > > > on skew, it could be done every (say) 10 seconds while allowing > > > 2 milliseconds skew instead of the current 100 microseconds. > > > > Yes, this can reduce the OS noise much. One problem is if we make it > > a general interface, there is some clocksource whose warp time is > > less than 10 seconds, like ACPI PM_TIMER (3-4 seconds), and I don't > > know if other ARCHs have similar cases. > > Maybe a simpler approach is for systems with such high sensitivity to > OS noise to simply disable the clocksource watchdog. ;-)
That's what the reported did, test with and without "tsc=reliable" parameter :)
And AFAIK, many customers with big server farms hate to add more cmdline parameters when we suggested so.
> > > > Is this about the concern of possible TSC frequency calibration > > > > issue, as the 40 ms per second drift between HPET and TSC? With > > > > b50db7095fe0 backported, we also have another patch to force TSC > > > > calibration for those platforms which get the TSC freq directly > > > > from CPUID or MSR and don't have such info in dmesg: > > > > "tsc: Refined TSC clocksource calibration: 2693.509 MHz" > > > > > > > > https://lore.kernel.org/lkml/20220509144110.9242-1-feng.tang@intel.com/ > > > > > > > > We did met tsc calibration issue due to some firmware issue, and > > > > this can help to catch it. You can try it if you think it's relevant. > > > > > > I am giving this a go, thank you! > > > > Thanks for spending time testing it! > > And here are the results from setting tsc_force_recalibrate to 1: > > $ dmesg | grep -E 'calibrat|clocksource' > [ 5.272939] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns > [ 16.830644] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns > [ 17.938020] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns > [ 24.548583] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns > [ 49.762432] clocksource: Switched to clocksource tsc-early > [ 50.076769] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns > [ 55.615946] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns > [ 55.640270] clocksource: Switched to clocksource tsc > [ 56.694371] tsc: Warning: TSC freq calibrated by CPUID/MSR differs from what is calibrated by HW timer, please check with vendor!! > [ 56.724550] tsc: Previous calibrated TSC freq: 1896.000 MHz > [ 56.737646] tsc: TSC freq recalibrated by [HPET]: 1975.000 MHz
Looks like there is really something wrong here. I assume the first number '1896 MHz' is got from CPUID(0x15)'s math calculation.
I thinks 2 more things could be try:
* add "nohpet" to the cmdline, so the tsc_force_recalibrate should use ACPI PM_TIMER to do the calibration, say a third-party check. * If the system don't have auto-adjusted time setting like NTP, I guess the system time will have obvious drift comparing to a normal clock or a mobile phone time, as the deviation is about 4%, which is 2.4 minutes per hour.
> Apologies for the delay, but reconfigurations put the system off the > net for some time.
No problem at all, it's your holiday time! Thanks for trying this!
Thanks, Feng
> Thanx, Paul
| |