Messages in this thread | | | Date | Fri, 13 Jan 2023 11:02:26 -0800 | From | Ricardo Neri <> | Subject | Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group |
| |
On Wed, Jan 11, 2023 at 04:04:23PM +0000, Valentin Schneider wrote: > On 28/12/22 20:00, Ricardo Neri wrote: > > On Thu, Dec 22, 2022 at 04:55:58PM +0000, Valentin Schneider wrote: > >> Some of this is new to me - I had missed the original series introducing > >> this. However ISTM that this is conflating two concepts: SMT occupancy > >> balancing, and asym packing. > >> > >> Take the !local_is_smt :: sg_busy_cpus >= 2 :: return true; path. It does > >> not involve asym packing priorities at all. This can end up in an > >> ASYM_PACKING load balance, > > > > Yes, this the desired result: an idle low-priority CPU should help a high- > > priority core with more than one busy SMT sibling. But yes, it does not > > relate to priorities and can be implemented differently. > > > >> which per calculate_imbalance() tries to move > >> *all* tasks to the higher priority CPU - in the case of SMT balancing, > >> we don't want to totally empty the core, just its siblings. > > > > But it will not empty the core, only one of its SMT siblings. A single > > sibling will be selected in find_busiest_queue(). The other siblings will > > be unaffected. > > > > Right > > >> > >> Is there an ITMT/big.LITTLE (or however x86 calls it) case that invalidates > >> the above? > > > > Please see below. > > > >> > >> Say, what's not sufficient with the below? AFAICT the only thing that isn't > >> covered is the sg_busy_cpus >= 2 thing, but IMO that's SMT balancing, not > >> asym packing - if the current calculate_imbalance() doesn't do it, it > >> should be fixed to do it. > > > > Agreed. > > > >>Looking at the > >> > >> local->group_type == group_has_spare > >> > >> case, it looks like it should DTRT. > > > > I had tried (and failed) to have find_busiest_group() handle the > > !local_is_smt :: sg_busy_cpus >= 2 case. Now I think I made it work. > > > > When the busiest group is not overloaded, find_busiest_group() and > > local->group = group_has_spare during an idle load balance events the > > number of *idle* CPUs. However, this does not work if the local and busiest > > groups have different weights. In SMT2, for instance, if busiest has 2 busy > > CPUs (i.e., 0 idle CPUs) and the destination CPU is idle, the difference in > > the number of idle CPUs is 1. find_busiest_group() will incorrectly goto > > out_balanced. > > > > This issue very visible in Intel hybrid processors because the big cores > > have SMT but the small cores do not. It can, however, be reproduced in non- > > hybrid processors by offlining the SMT siblings of some cores. > > > > I think I follow. If we're comparing two groups each spanning an SMT2 core, > then > > busiest->group_weight > 1 && local->idle_cpus <= (busiest->idle_cpus + 1) > > is false if local is fully idle and busiest fully busy, but if local > becomes a non-SMT core, then that's true and we goto out_balanced.
Exactly right.
> > > With that said, shouldn't SD_PREFER_SIBLING help here? cf. > > if (sds.prefer_sibling && local->group_type == group_has_spare && > busiest->sum_nr_running > local->sum_nr_running + 1)
It does not help because sds.prefer_sibling is false: an non-SMT core is looking into offloading a fully_busy SMT core at the "MC" domain. sds.prefer_sibling is set in update_sd_lb_stats() if the sched domain's child has the SD_PREFER_SIBLING flag. Since the destination CPU is the non-SMT core, there is no child.
> > It should be set on any topology level below the NUMA ones, we do remove it > on SD_ASYM_CPUCAPACITY levels because this used to interfere with misfit > balancing (it would override the group_type), things are a bit different > since Vincent's rewrite of load_balance() but I think we still want it off > there.
I see in find_busiest_group() that group_misfit_task is evaluated before sds.prefer_sibling.
> I would expect it to be set in your system, though whether this is > playing nice with the asymmetry is another matter :-)
I recall a few instances of SD_PREFER_SIBLING causing trouble me, but I need to investigate more.
Thanks and BR, Ricardo
| |