Messages in this thread | | | Date | Tue, 2 Mar 2021 11:56:20 +0100 | From | Peter Zijlstra <> | Subject | Re: [PATCH v4] sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2 |
| |
On Wed, Feb 24, 2021 at 04:09:44PM +1300, Barry Song wrote: > As long as NUMA diameter > 2, building sched_domain by sibling's child > domain will definitely create a sched_domain with sched_group which will > span out of the sched_domain: > > +------+ +------+ +-------+ +------+ > | node | 12 |node | 20 | node | 12 |node | > | 0 +---------+1 +--------+ 2 +-------+3 | > +------+ +------+ +-------+ +------+ > > domain0 node0 node1 node2 node3 > > domain1 node0+1 node0+1 node2+3 node2+3 > + > domain2 node0+1+2 | > group: node0+1 | > group:node2+3 <-------------------+ > > when node2 is added into the domain2 of node0, kernel is using the child > domain of node2's domain2, which is domain1(node2+3). Node 3 is outside > the span of the domain including node0+1+2. > > This will make load_balance() run based on screwed avg_load and group_type > in the sched_group spanning out of the sched_domain, and it also makes > select_task_rq_fair() pick an idle CPU outside the sched_domain. > > Real servers which suffer from this problem include Kunpeng920 and 8-node > Sun Fire X4600-M2, at least. > > Here we move to use the *child* domain of the *child* domain of node2's > domain2 as the new added sched_group. At the same, we re-use the lower > level sgc directly. > +------+ +------+ +-------+ +------+ > | node | 12 |node | 20 | node | 12 |node | > | 0 +---------+1 +--------+ 2 +-------+3 | > +------+ +------+ +-------+ +------+ > > domain0 node0 node1 +- node2 node3 > | > domain1 node0+1 node0+1 | node2+3 node2+3 > | > domain2 node0+1+2 | > group: node0+1 | > group:node2 <-------------------+ > > While the lower level sgc is re-used, this patch only changes the remote > sched_groups for those sched_domains playing grandchild trick, therefore, > sgc->next_update is still safe since it's only touched by CPUs that have > the group span as local group. And sgc->imbalance is also safe because > sd_parent remains the same in load_balance and LB only tries other CPUs > from the local group. > Moreover, since local groups are not touched, they are still getting > roughly equal size in a TL. And should_we_balance() only matters with > local groups, so the pull probability of those groups are still roughly > equal. >
> Reported-by: Valentin Schneider <valentin.schneider@arm.com> > Tested-by: Meelis Roos <mroos@linux.ee> > Reviewed-by: Valentin Schneider <valentin.schneider@arm.com> > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
Thanks!
| |