Messages in this thread Patch in this message | | | From | Nicolas Saenz Julienne <> | Subject | [PATCH 5/6] mm/page_alloc: Allow remote per-cpu page list draining | Date | Tue, 21 Sep 2021 18:13:23 +0200 |
| |
Some setups, notably NOHZ_FULL CPUs, are too busy to handle the per-cpu drain work queued by __drain_all_pages(). So introduce new a mechanism to remotely drain the per-cpu lists. It is made possible by using a more constraining locking scheme, which is disabled by default and can be enabled through the 'remote_pcpu_cache_access' static key. Regular users shouldn't see any functional or performance change. Upcoming patches will make use of the static key.
This is based on previous work by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1].
[1] https://patchwork.kernel.org/project/linux-mm/patch/20190424111208.24459-3-bigeasy@linutronix.de/ Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com> --- mm/page_alloc.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3244eb2ab51b..717df675ea06 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3268,15 +3268,25 @@ static void __drain_all_pages(struct zone *zone, bool force_all_cpus) cpumask_clear_cpu(cpu, &cpus_with_pcps); } - for_each_cpu(cpu, &cpus_with_pcps) { - struct pcpu_drain *drain = per_cpu_ptr(&pcpu_drain, cpu); + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + for_each_cpu(cpu, &cpus_with_pcps) { + if (zone) { + drain_pages_zone(cpu, zone); + } else { + drain_pages(cpu); + } + } + } else { + for_each_cpu(cpu, &cpus_with_pcps) { + struct pcpu_drain *drain = per_cpu_ptr(&pcpu_drain, cpu); - drain->zone = zone; - INIT_WORK(&drain->work, drain_local_pages_wq); - queue_work_on(cpu, mm_percpu_wq, &drain->work); + drain->zone = zone; + INIT_WORK(&drain->work, drain_local_pages_wq); + queue_work_on(cpu, mm_percpu_wq, &drain->work); + } + for_each_cpu(cpu, &cpus_with_pcps) + flush_work(&per_cpu_ptr(&pcpu_drain, cpu)->work); } - for_each_cpu(cpu, &cpus_with_pcps) - flush_work(&per_cpu_ptr(&pcpu_drain, cpu)->work); mutex_unlock(&pcpu_drain_mutex); } -- 2.31.1
| |