lkml.org 
[lkml]   [2019]   [Jul]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH v3] sched/topology: Improve load balancing on AMD EPYC
    On Tue, Jul 23, 2019 at 11:48:30AM +0100, Matt Fleming wrote:
    > SD_BALANCE_{FORK,EXEC} and SD_WAKE_AFFINE are stripped in sd_init()
    > for any sched domains with a NUMA distance greater than 2 hops
    > (RECLAIM_DISTANCE). The idea being that it's expensive to balance
    > across domains that far apart.
    >
    > However, as is rather unfortunately explained in
    >
    > commit 32e45ff43eaf ("mm: increase RECLAIM_DISTANCE to 30")
    >
    > the value for RECLAIM_DISTANCE is based on node distance tables from
    > 2011-era hardware.
    >
    > Current AMD EPYC machines have the following NUMA node distances:
    >
    > node distances:
    > node 0 1 2 3 4 5 6 7
    > 0: 10 16 16 16 32 32 32 32
    > 1: 16 10 16 16 32 32 32 32
    > 2: 16 16 10 16 32 32 32 32
    > 3: 16 16 16 10 32 32 32 32
    > 4: 32 32 32 32 10 16 16 16
    > 5: 32 32 32 32 16 10 16 16
    > 6: 32 32 32 32 16 16 10 16
    > 7: 32 32 32 32 16 16 16 10
    >
    > where 2 hops is 32.
    >
    > The result is that the scheduler fails to load balance properly across
    > NUMA nodes on different sockets -- 2 hops apart.
    >
    > For example, pinning 16 busy threads to NUMA nodes 0 (CPUs 0-7) and 4
    > (CPUs 32-39) like so,
    >
    > $ numactl -C 0-7,32-39 ./spinner 16
    >
    > causes all threads to fork and remain on node 0 until the active
    > balancer kicks in after a few seconds and forcibly moves some threads
    > to node 4.
    >
    > Override node_reclaim_distance for AMD Zen.
    >
    > Signed-off-by: Matt Fleming <matt@codeblueprint.co.uk>
    > Cc: "Suthikulpanit, Suravee" <Suravee.Suthikulpanit@amd.com>
    > Cc: Mel Gorman <mgorman@techsingularity.net>
    > Cc: "Lendacky, Thomas" <Thomas.Lendacky@amd.com>
    > Cc: Borislav Petkov <bp@alien8.de>

    Acked-by: Mel Gorman <mgorman@techsingularity.net>

    The only caveat I can think of is that a future generation of Zen might
    take a different magic number than 32 as their remote distance. If or
    when this happens, it'll need additional smarts but lacking a crystal
    ball, we can cross that bridge when we come to it.

    --
    Mel Gorman
    SUSE Labs

    \
     
     \ /
      Last update: 2019-07-23 13:43    [W:2.229 / U:0.168 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site