Messages in this thread | | | Date | Mon, 28 Mar 2022 17:17:46 +0200 | From | Peter Zijlstra <> | Subject | Re: [External] Re: [PATCH] sched/fair: fix broken bandwidth control with nohz_full |
| |
On Mon, Mar 28, 2022 at 09:50:05PM +0800, Chengming Zhou wrote: > On 2022/3/28 21:20, Peter Zijlstra wrote: > > On Mon, Mar 28, 2022 at 07:07:51PM +0800, Chengming Zhou wrote: > >> With nohz_full enabled on cpu, the scheduler_tick() will be stopped > >> when only one CFS task left on rq. > >> > >> scheduler_tick() > >> task_tick_fair() > >> entity_tick() > >> update_curr() > >> account_cfs_rq_runtime(cfs_rq, delta_exec) --> stopped > >> > >> So that running task can't account its runtime periodically, but > >> the cfs_bandwidth hrtimer still __refill_cfs_bandwidth_runtime() > >> periodically. Later in one period, the task would account very > >> big delta_exec, which cause the cfs_rq to be throttled for a > >> long time. > >> > >> There are two solutions for the problem, the first is that we > >> can check in sched_can_stop_tick() if current task's cfs_rq > >> have runtime_enabled, in which case we don't stop tick. But > >> it will make nohz_full almost useless in cloud environment > >> that every container has the cpu bandwidth control setting. > > > > How is NOHZ_FULL useful in that environment to begin with? If you set > > bandwidth crap, the expectation is that there is overcommit, which more > > or less assumes lots of scheduling, presumably VMs or somesuch crud. > > > > So how does NOHZ_FULL make sense? > > Yes, we have scheduled some VMs in cgroups on the host, which > enabled NOHZ_FULL to reduce the interference of tick to vcpu task > if it's the only task running on cpu. > > This problem will however throttle it wrongly, even if it hasn't > used up its quota. > > Do you suggest that we shouldn't stop tick when the current task's > cfs_rq has runtime_enabled ?
I'm not suggesting anything just yet as I'm not sure I understand things well enough. I'm just wondering if NOHZ_FULL makes sense for you since NOHZ_FULL makes system entry/exit so much more expensive.
NOHZ_FULL is for use-cases that 'never' intend to go into the kernel, your use-case actively relies on going into the kernel. Hence the confusion.
| |