lkml.org 
[lkml]   [2021]   [Jan]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRE: 5.11-rc4+git: Shortest NUMA path spans too many nodes
Date


> -----Original Message-----
> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com]
> Sent: Friday, January 22, 2021 11:05 PM
> To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>; Valentin Schneider
> <valentin.schneider@arm.com>; Meelis Roos <mroos@linux.ee>; LKML
> <linux-kernel@vger.kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>; Vincent Guittot
> <vincent.guittot@linaro.org>; Mel Gorman <mgorman@suse.de>
> Subject: Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes
>
> On 21/01/2021 22:17, Song Bao Hua (Barry Song) wrote:
> >
> >
> >> -----Original Message-----
> >> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com]
> >> Sent: Friday, January 22, 2021 7:54 AM
> >> To: Valentin Schneider <valentin.schneider@arm.com>; Meelis Roos
> >> <mroos@linux.ee>; LKML <linux-kernel@vger.kernel.org>
> >> Cc: Peter Zijlstra <peterz@infradead.org>; Vincent Guittot
> >> <vincent.guittot@linaro.org>; Song Bao Hua (Barry Song)
> >> <song.bao.hua@hisilicon.com>; Mel Gorman <mgorman@suse.de>
> >> Subject: Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes
> >>
> >> On 21/01/2021 19:21, Valentin Schneider wrote:
> >>> On 21/01/21 19:39, Meelis Roos wrote:
>
> [...]
>
> >> # cat /sys/devices/system/node/node*/distance
> >> 10 12 12 14 14 14 14 16
> >> 12 10 14 12 14 14 12 14
> >> 12 14 10 14 12 12 14 14
> >> 14 12 14 10 12 12 14 14
> >> 14 14 12 12 10 14 12 14
> >> 14 14 12 12 14 10 14 12
> >> 14 12 14 14 12 14 10 12
> >> 16 14 14 14 14 12 12 10
> >>
> >> The '16' seems to be the culprit. How does such a topo look like?
>
> Maybe like this:
>
> _________
> | |
> .-6 0 4-.
> | \ / \ / |
> | 1 2 |
> | \ \ |
> --7 3----5 |
> | |____|_|
> |_______|
>
> >
> > Once we get a topology like this:
> >
> >
> > +------+ +------+ +-------+ +------+
> > | node | |node | | node | |node |
> > | +---------+ +--------+ +-------+ |
> > +------+ +------+ +-------+ +------+
> >
> > We can reproduce this issue.
> > For example, every cpu with the below numa_distance can have
> > "groups don't span domain->span":
> > node 0 1 2 3
> > 0: 10 12 20 22
> > 1: 12 10 22 24
> > 2: 20 22 10 12
> > 3: 22 24 12 10
> 2 20 2
> So this should look like: 1 --- 0 ---- 2 --- 3

Yes. So here we are facing another problem:
kernel/sched/topology.c has an assumption that:
node_distance(0,j) includes all distances in
node_distance(i,j).

void sched_init_numa(void)
{
...
*
* Assumes node_distance(0,j) includes all distances in
* node_distance(i,j) in order to avoid cubic time.
*/
next_distance = curr_distance;
for (i = 0; i < nr_node_ids; i++) {
for (j = 0; j < nr_node_ids; j++) {
for (k = 0; k < nr_node_ids; k++)
}

but obviously we are not this case. Right now, we are getting
some performance decrease due to this, probably I'll start another
thread for it.

Thanks
Barry

\
 
 \ /
  Last update: 2021-01-22 12:55    [W:0.048 / U:0.424 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site