Messages in this thread | | | Date | Tue, 28 Jun 2022 15:32:46 +0530 | Subject | Re: [PATCH] srcu: Reduce blocking agressiveness of expedited grace periods further | From | Neeraj Upadhyay <> |
| |
On 6/28/2022 3:01 PM, Marc Zyngier wrote: > On Tue, 28 Jun 2022 10:17:24 +0100, > Neeraj Upadhyay <quic_neeraju@quicinc.com> wrote: >> >> >> >> On 6/28/2022 2:32 PM, Marc Zyngier wrote: >>> On Mon, 27 Jun 2022 13:37:06 +0100, >>> Neeraj Upadhyay <quic_neeraju@quicinc.com> wrote: >>>> >>>> Commit 640a7d37c3f4 ("srcu: Block less aggressively for expedited >>>> grace periods") highlights a problem where aggressively blocking >>>> SRCU expedited grace periods, as was introduced in commit >>>> 282d8998e997 ("srcu: Prevent expedited GPs and blocking readers >>>> from consuming CPU"), introduces ~2 minutes delay to the overall >>>> ~3.5 minutes boot time, when starting VMs with "-bios QEMU_EFI.fd" >>>> cmdline on qemu, which results in very high rate of memslots >>>> add/remove, which causes > ~6000 synchronize_srcu() calls for >>>> kvm->srcu SRCU instance. >>>> >>>> Below table captures the experiments done by Zhangfei Gao, Shameer, >>>> to measure the boottime impact with various values of non-sleeping >>>> per phase counts, with HZ_250 and preemption enabled: >>>> >>>> +──────────────────────────+────────────────+ >>>> | SRCU_MAX_NODELAY_PHASE | Boot time (s) | >>>> +──────────────────────────+────────────────+ >>>> | 100 | 30.053 | >>>> | 150 | 25.151 | >>>> | 200 | 20.704 | >>>> | 250 | 15.748 | >>>> | 500 | 11.401 | >>>> | 1000 | 11.443 | >>>> | 10000 | 11.258 | >>>> | 1000000 | 11.154 | >>>> +──────────────────────────+────────────────+ >>>> >>>> Analysis on the experiment results showed improved boot time >>>> with non blocking delays close to one jiffy duration. This >>>> was also seen when number of per-phase iterations were scaled >>>> to one jiffy. >>>> >>>> So, this change scales per-grace-period phase number of non-sleeping >>>> polls, soiuch that, non-sleeping polls are done for one jiffy. In addition >>>> to this, srcu_get_delay() call in srcu_gp_end(), which is used to calculate >>>> the delay used for scheduling callbacks, is replaced with the check for >>>> expedited grace period. This is done, to schedule cbs for completed expedited >>>> grace periods immediately, which results in improved boot time seen in >>>> experiments. >>>> >>>> In addition to the changes to default per phase delays, this change >>>> adds 3 new kernel parameters - srcutree.srcu_max_nodelay, >>>> srcutree.srcu_max_nodelay_phase, srcutree.srcu_retry_check_delay. >>>> This allows users to configure the srcu grace period scanning delays, >>>> depending on their system configuration requirements. >>>> >>>> Signed-off-by: Neeraj Upadhyay <quic_neeraju@quicinc.com> >>> >>> I've given this a go on one of my test platforms (the one I noticed >>> the issue on the first place), and found that the initial part of the >>> EFI boot under KVM (pointlessly wiping the emulated flash) went down >>> to 1m7s from 3m50s (HZ=250). >>> >>> Clearly a massive improvement, but still a far cry from the original >>> ~40s (yes, this box is utter crap -- which is why I use it). >> >> Do you see any improvement by using "srcutree.srcu_max_nodelay=1000" >> bootarg, on top of this patch? > > Yup, this brings it back to 43s on a quick test run, which is close > enough to what I had before. >
Cool, thanks!
> How does a random user come up with such a value though? >
It need to be tuned :) The patch actually adds 2 jiffies (vs the one jiffy mentioned in the commit log) of non-sleep per phase delay. Each phase iteration uses a delay of 10 us. So, for CONFIG_HZ_250 its around 800 iterations.
Thanks Neeraj
> Thanks, > > M. >
| |