Messages in this thread | | | Date | Tue, 28 Jun 2022 06:46:59 -0700 | From | "Paul E. McKenney" <> | Subject | Re: [PATCH] srcu: Reduce blocking agressiveness of expedited grace periods further |
| |
On Tue, Jun 28, 2022 at 10:31:54AM +0100, Marc Zyngier wrote: > On Tue, 28 Jun 2022 10:17:24 +0100, > Neeraj Upadhyay <quic_neeraju@quicinc.com> wrote: > > > > > > > > On 6/28/2022 2:32 PM, Marc Zyngier wrote: > > > On Mon, 27 Jun 2022 13:37:06 +0100, > > > Neeraj Upadhyay <quic_neeraju@quicinc.com> wrote: > > >> > > >> Commit 640a7d37c3f4 ("srcu: Block less aggressively for expedited > > >> grace periods") highlights a problem where aggressively blocking > > >> SRCU expedited grace periods, as was introduced in commit > > >> 282d8998e997 ("srcu: Prevent expedited GPs and blocking readers > > >> from consuming CPU"), introduces ~2 minutes delay to the overall > > >> ~3.5 minutes boot time, when starting VMs with "-bios QEMU_EFI.fd" > > >> cmdline on qemu, which results in very high rate of memslots > > >> add/remove, which causes > ~6000 synchronize_srcu() calls for > > >> kvm->srcu SRCU instance. > > >> > > >> Below table captures the experiments done by Zhangfei Gao, Shameer, > > >> to measure the boottime impact with various values of non-sleeping > > >> per phase counts, with HZ_250 and preemption enabled: > > >> > > >> +──────────────────────────+────────────────+ > > >> | SRCU_MAX_NODELAY_PHASE | Boot time (s) | > > >> +──────────────────────────+────────────────+ > > >> | 100 | 30.053 | > > >> | 150 | 25.151 | > > >> | 200 | 20.704 | > > >> | 250 | 15.748 | > > >> | 500 | 11.401 | > > >> | 1000 | 11.443 | > > >> | 10000 | 11.258 | > > >> | 1000000 | 11.154 | > > >> +──────────────────────────+────────────────+ > > >> > > >> Analysis on the experiment results showed improved boot time > > >> with non blocking delays close to one jiffy duration. This > > >> was also seen when number of per-phase iterations were scaled > > >> to one jiffy. > > >> > > >> So, this change scales per-grace-period phase number of non-sleeping > > >> polls, soiuch that, non-sleeping polls are done for one jiffy. In addition > > >> to this, srcu_get_delay() call in srcu_gp_end(), which is used to calculate > > >> the delay used for scheduling callbacks, is replaced with the check for > > >> expedited grace period. This is done, to schedule cbs for completed expedited > > >> grace periods immediately, which results in improved boot time seen in > > >> experiments. > > >> > > >> In addition to the changes to default per phase delays, this change > > >> adds 3 new kernel parameters - srcutree.srcu_max_nodelay, > > >> srcutree.srcu_max_nodelay_phase, srcutree.srcu_retry_check_delay. > > >> This allows users to configure the srcu grace period scanning delays, > > >> depending on their system configuration requirements. > > >> > > >> Signed-off-by: Neeraj Upadhyay <quic_neeraju@quicinc.com> > > > > > > I've given this a go on one of my test platforms (the one I noticed > > > the issue on the first place), and found that the initial part of the > > > EFI boot under KVM (pointlessly wiping the emulated flash) went down > > > to 1m7s from 3m50s (HZ=250). > > > > > > Clearly a massive improvement, but still a far cry from the original > > > ~40s (yes, this box is utter crap -- which is why I use it). > > > > Do you see any improvement by using "srcutree.srcu_max_nodelay=1000" > > bootarg, on top of this patch? > > Yup, this brings it back to 43s on a quick test run, which is close > enough to what I had before. > > How does a random user come up with such a value though?
There was some talk of moving from synchronize_srcu_expedited() to call_srcu() with the occasional srcu_barrier() to avoid OOM. If that proves to be practical, that should get decent performance with little tuning. But in the meantime, we need to avoid hangs due to CPU-bound tasks in one workload while still avoiding massive boot-time slowdowns in your workload.
Right now, Neeraj's carefully tuned approach is the one way we know to square this particular circle.
Thanx, Paul
| |