Messages in this thread Patch in this message | | | From | Srikar Dronamraju <> | Subject | [PATCH 05/10] sched/fair: Call wake_affine only if necessary | Date | Thu, 22 Apr 2021 15:53:21 +0530 |
| |
Currently if a waker is the only running thread of the CPU, and waking a wakee (both of them interacting over a pipe aka sync wakeups) with the wakee previously running on a different LLC), then wakee is pulled and queued on the CPU that is running the waker.
This is true even if the previous CPU was completely idle. The rationale being waker would soon relinquish the CPU and wakee would benefit from the cache access. However if the previous CPU is idle, then wakee thread will incur latency cost of migration to the waker CPU + latency cost of the waker context switching.
Before the waker switches out, load balancer could also kick in and pull the wakee out resulting in another migration cost. This will mostly hurt systems where LLC have just one core. For example:- Hackbench. Both the threads of hackbench would then end up running on the same core resulting in CPUs running in higher SMT modes, more load balances and non optimal performance.
It would be beneficial to run the wakee thread on the same CPU as waker only if other CPUs are busy. To achieve this move this part of the code that check if waker is the only running thread to early part of wake_affine_weight().
Also post this change, wake_affine_idle() will only look at wakeups within the LLC. For wakeups within LLC, there should be no difference between sync and no-sync. For wakeups across LLC boundaries, lets use wake_affine_idler_llc.
Cc: LKML <linux-kernel@vger.kernel.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Gautham R Shenoy <ego@linux.vnet.ibm.com> Cc: Parth Shah <parth@linux.ibm.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Valentin Schneider <valentin.schneider@arm.com> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Rik van Riel <riel@surriel.com> Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com> --- kernel/sched/fair.c | 22 +++++++++++----------- kernel/sched/features.h | 2 +- 2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 943621367a96..f8c98e544418 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5799,8 +5799,7 @@ static int wake_wide(struct task_struct *p) * scheduling latency of the CPUs. This seems to work * for the overloaded case. */ -static int -wake_affine_idle(int this_cpu, int prev_cpu, int sync) +static int wake_affine_idle(int this_cpu, int prev_cpu) { /* * If this_cpu is idle, it implies the wakeup is from interrupt @@ -5814,15 +5813,12 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync) * a cpufreq perspective, it's better to have higher utilisation * on one CPU. */ - if (available_idle_cpu(this_cpu) && cpus_share_cache(this_cpu, prev_cpu)) - return available_idle_cpu(prev_cpu) ? prev_cpu : this_cpu; + if (available_idle_cpu(prev_cpu) || sched_idle_cpu(prev_cpu)) + return prev_cpu; - if (sync && cpu_rq(this_cpu)->nr_running == 1) + if (available_idle_cpu(this_cpu) || sched_idle_cpu(this_cpu)) return this_cpu; - if (available_idle_cpu(prev_cpu)) - return prev_cpu; - return nr_cpumask_bits; } @@ -5838,6 +5834,9 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p, if (sync) { unsigned long current_load = task_h_load(current); + if (cpu_rq(this_cpu)->nr_running <= 1) + return this_cpu; + if (current_load > this_eff_load) return this_cpu; @@ -5933,12 +5932,13 @@ static int wake_affine_idler_llc(struct task_struct *p, int this_cpu, int prev_c static int wake_affine(struct sched_domain *sd, struct task_struct *p, int this_cpu, int prev_cpu, int sync) { + bool share_caches = cpus_share_cache(prev_cpu, this_cpu); int target = nr_cpumask_bits; - if (sched_feat(WA_IDLE)) - target = wake_affine_idle(this_cpu, prev_cpu, sync); + if (sched_feat(WA_IDLE) && share_caches) + target = wake_affine_idle(this_cpu, prev_cpu); - if (sched_feat(WA_IDLER_LLC) && target == nr_cpumask_bits) + else if (sched_feat(WA_IDLER_LLC) && !share_caches) target = wake_affine_idler_llc(p, this_cpu, prev_cpu, sync); if (sched_feat(WA_WEIGHT) && target == nr_cpumask_bits) diff --git a/kernel/sched/features.h b/kernel/sched/features.h index c77349a47e01..c60a6d2b2126 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -83,7 +83,7 @@ SCHED_FEAT(ATTACH_AGE_LOAD, true) SCHED_FEAT(WA_IDLE, true) SCHED_FEAT(WA_WEIGHT, true) -SCHED_FEAT(WA_IDLER_LLC, false) +SCHED_FEAT(WA_IDLER_LLC, true) SCHED_FEAT(WA_BIAS, true) /* -- 2.18.2
| |