lkml.org 
[lkml]   [2022]   [Feb]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC PATCH 0/5] introduce sched-idle balancing
    On Thu, Feb 17, 2022 at 11:43:56PM +0800, Abel Wu wrote:
    > Current load balancing is mainly based on cpu capacity
    > and task util, which makes sense in the POV of overall
    > throughput. While there still might be some improvement
    > can be done by reducing number of overloaded cfs rqs if
    > sched-idle or idle rq exists.
    >
    > An CFS runqueue is considered overloaded when there are
    > more than one pullable non-idle tasks on it (since sched-
    > idle cpus are treated as idle cpus). And idle tasks are
    > counted towards rq->cfs.idle_h_nr_running, that is either
    > assigned SCHED_IDLE policy or placed under idle cgroups.
    >

    It's not clear how your tests evaluated the balancing of SCHED_IDLE tasks
    versus the existing idle balancing and isolated that impact. I suspect
    the tests may primarily measured the effect of the SIS filter.

    > So in short, the goal of the sched-idle balancing is to
    > let the *non-idle tasks* make full use of cpu resources.
    > To achieve that, we mainly do two things:
    >
    > - pull non-idle tasks for sched-idle or idle rqs
    > from the overloaded ones, and
    >
    > - prevent pulling the last non-idle task in an rq
    >
    > The mask of overloaded cpus is updated in periodic tick
    > and the idle path at the LLC domain basis. This cpumask
    > will also be used in SIS as a filter, improving idle cpu
    > searching.
    >

    As the overloaded mask may be updated on each idle, it could be a
    significant source of cache misses between CPUs sharing the domain for
    workloads that rapidly idle so there should be data on whether cache misses
    are increased heavily. It also potentially delays the CPU reaching idle
    but it may not be by much.

    The filter may be out of date. It takes up to one tick to detect
    overloaded and the filter to have a positive impact. As a CPU is not
    guaranteed to enter idle if there is at least one CPU-bound task, it may
    also be up to 1 tick before the mask is cleared. I'm not sure this is a
    serious problem though as SIS would not pick the CPU with the CPU-bound
    task anyway.

    At minimum, the filter should be split out and considered first as it
    is the most likely reason why a performance difference was measured. It
    has some oddities like why nr_overloaded is really a boolean and as
    it's under rq lock, it's not clear why it's atomic. The changelog
    would ideally contain some comment on the impact to cache misses
    if any and some sort of proof that SIS search depth is reduced which
    https://lore.kernel.org/lkml/20210726102247.21437-2-mgorman@techsingularity.net/
    may be some help.

    At that point, compare the idle task balancing on top to isolate how
    much it improves things if any and identify why existing balancing is
    insufficient. Split out the can_migrate_task change beforehand in case it
    is the main source of difference as opposed to the new balancing mechanism.

    --
    Mel Gorman
    SUSE Labs

    \
     
     \ /
      Last update: 2022-02-24 17:48    [W:4.408 / U:0.020 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site