Messages in this thread | | | Date | Tue, 19 Sep 2023 17:01:38 +0800 | From | Chen Yu <> | Subject | Re: [RFC PATCH 2/2] sched/fair: skip the cache hot CPU in select_idle_cpu() |
| |
Hi Gautham,
Sorry for late reply,
On 2023-09-15 at 20:48:14 +0530, Gautham R. Shenoy wrote: > Hello Chen Yu, > > On Thu, Sep 14, 2023 at 08:09:26PM +0800, Chen Yu wrote: > [..snip..] > > > > > > > So despite "reserving" the CPU for p1, which is likely to have its > > > data still hot in the case, we would have scheduled p1', thus > > > defeating the whole purpose of reservation. > > > > > > > I see. So you mean, although we reserve the CPU for the wakee, > > the wakee might not choose its previous CPU, which is against our > > goal. > > > Yes, but only because some other task could have run on the previous > CPU. That other task could be something that was woken up on that CPU > due to: > > 1) wake-affine choosing that CPU > 2) newidle-balance pulling the other task on that CPU > 3) !wake-affine && that CPU was also the other task's previous CPU > > It could also be due to this wakee task being woken up on the waker > CPU due to wake-affine. > > > > > The reason to prevent the wakee choosing its previous CPU could be: > > 1. wake_affine() choose the waker's CPU rather the wakee's previous CPU, or > > 2. the wakee's CPU has already been taken by someone else, via newidle_balance(). > > > > > > For 1, I think Prateek has expressed the concern. One mitigation method could be > > that, we give penalty to that wakee, if it decides not to choose its previous CPU: > > We would be penalizing the task for something that the scheduler > decides :-) > > As you point out below, in the presence of the WF_SYNC flag, > wake_affine_idle() prefer the waker CPU over the previous CPU when > they are on different LLCs and when the waker is the only task. > > This strategy makes sense for two reasons: > > 1) The wakee may be consuming the data produced by the waker. > 2) Since the wakeup will happen on the local CPU, there is no risk of > task-stacking, exactly what your SIS_CURRENT patchset was > attempting. > > But this strategy would also result in increased task-migration. Which > both Mattieu and you have found is not so beneficial for workloads > such as hackbench. Is it only because task's data is still hot in the > previous CPU's cache ? Or is there more to it ? > > > It would be good to confirm if this is why lower migration is better > for these kinds of workloads. >
Right. According to the previous hackbench test for shared runqueue, higher migration brings higher DSB miss rate [1]. I'll collect some statistics with this patch applied to confirm.
https://lore.kernel.org/lkml/ZO7e5YaS71cXVxQN@chenyu5-mobl2/ > > > > " > > new_cpu = select_idle_sibling(p, prev_cpu, new_cpu); > > if (new_cpu != prev_cpu) > > p->burst_sleep_avg >>= 1; > > So the duration of reservation could be shrinked. > > " > > > > For 2, maybe inhit the newidle balance, something in my mind: > > > > > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -12022,6 +12022,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) > > u64 t0, t1, curr_cost = 0; > > struct sched_domain *sd; > > int pulled_task = 0; > > + bool cache_hot = false; > > > > update_misfit_status(NULL, this_rq); > > > > @@ -12055,8 +12056,19 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) > > rcu_read_lock(); > > sd = rcu_dereference_check_sched_domain(this_rq->sd); > > > > + if (sched_feat(SIS_CACHE)) { > > + s64 delta = this_rq->cache_hot_timeout - sched_clock_cpu(this_cpu); > > + > > + /* > > + * If a short time later, a short sleeping task will be woken up > > + * on this idle CPU, do not launch the newidle balance. > > + */ > > + if (delta > 0 && delta < this_rq->max_idle_balance_cost) > > + cache_hot = true; > > + } > > + > > if (!READ_ONCE(this_rq->rd->overload) || > > - (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) { > > + (sd && this_rq->avg_idle < sd->max_newidle_lb_cost) || cache_hot) { > > > > > if (sd) > > update_next_balance(sd, &next_balance); > > If the benefit that the workload obtains is really due to the data > being hot near its previous CPU, then this seems a sensible thing to > do. > > It would be good to confirm this. Let me get some IBS data for > hackbench which is the workload which likes a sticky wakeup. >
I'll collect the statistics too. Thanks for your time.
thanks, Chenyu
| |