Messages in this thread Patch in this message | | | Date | Tue, 19 Dec 2023 16:18:18 -0800 | Subject | [PATCH v7 07/23] locking/mutex: Switch to mutex handoffs for CONFIG_SCHED_PROXY_EXEC | From | John Stultz <> |
| |
From: Peter Zijlstra <peterz@infradead.org>
Since with SCHED_PROXY_EXEC, we will want to hand off locks to the tasks we are running on behalf of, switch to using mutex handoffs.
Cc: Joel Fernandes <joelaf@google.com> Cc: Qais Yousef <qyousef@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Ben Segall <bsegall@google.com> Cc: Zimuzo Ezeozue <zezeozue@google.com> Cc: Youssef Esmat <youssefesmat@google.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Daniel Bristot de Oliveira <bristot@redhat.com> Cc: Will Deacon <will@kernel.org> Cc: Waiman Long <longman@redhat.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: "Paul E. McKenney" <paulmck@kernel.org> Cc: Metin Kaya <Metin.Kaya@arm.com> Cc: Xuewen Yan <xuewen.yan94@gmail.com> Cc: K Prateek Nayak <kprateek.nayak@amd.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: kernel-team@android.com Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> [rebased, added comments and changelog] Signed-off-by: Juri Lelli <juri.lelli@redhat.com> [Fixed rebase conflicts] [squashed sched: Ensure blocked_on is always guarded by blocked_lock] Signed-off-by: Valentin Schneider <valentin.schneider@arm.com> [fix rebase conflicts, various fixes & tweaks commented inline] [squashed sched: Use rq->curr vs rq->proxy checks] Signed-off-by: Connor O'Brien <connoro@google.com> [jstultz: Split out only the very basic initial framework for proxy logic from a larger patch.] Signed-off-by: John Stultz <jstultz@google.com> --- v5: * Split out from core proxy patch v6: * Rework to use sched_proxy_exec() instead of #ifdef CONFIG_PROXY_EXEC v7: * Avoid disabling optimistic spinning at compile time so booting with sched_proxy_exec=off matches prior performance * Add comment in mutex-design.rst as suggested by Metin Kaya --- Documentation/locking/mutex-design.rst | 3 ++ kernel/locking/mutex.c | 42 +++++++++++++++----------- 2 files changed, 28 insertions(+), 17 deletions(-)
diff --git a/Documentation/locking/mutex-design.rst b/Documentation/locking/mutex-design.rst index 78540cd7f54b..57a5cb03f409 100644 --- a/Documentation/locking/mutex-design.rst +++ b/Documentation/locking/mutex-design.rst @@ -61,6 +61,9 @@ taken, depending on the state of the lock: waiting to spin on mutex owner, only to go directly to slowpath upon obtaining the MCS lock. + NOTE: Optimistic spinning will be avoided when using proxy execution + (SCHED_PROXY_EXEC) as we want to hand the lock off to the task that was + boosting the current owner. (iii) slowpath: last resort, if the lock is still unable to be acquired, the task is added to the wait-queue and sleeps until woken up by the diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 6084470773f6..11dc5cb7a5a3 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -416,6 +416,9 @@ static __always_inline bool mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx, struct mutex_waiter *waiter) { + if (sched_proxy_exec()) + return false; + if (!waiter) { /* * The purpose of the mutex_can_spin_on_owner() function is @@ -914,26 +917,31 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne mutex_release(&lock->dep_map, ip); - /* - * Release the lock before (potentially) taking the spinlock such that - * other contenders can get on with things ASAP. - * - * Except when HANDOFF, in that case we must not clear the owner field, - * but instead set it to the top waiter. - */ - owner = atomic_long_read(&lock->owner); - for (;;) { - MUTEX_WARN_ON(__owner_task(owner) != current); - MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP); - - if (owner & MUTEX_FLAG_HANDOFF) - break; + if (sched_proxy_exec()) { + /* Always force HANDOFF for Proxy Exec for now. Revisit. */ + owner = MUTEX_FLAG_HANDOFF; + } else { + /* + * Release the lock before (potentially) taking the spinlock + * such that other contenders can get on with things ASAP. + * + * Except when HANDOFF, in that case we must not clear the + * owner field, but instead set it to the top waiter. + */ + owner = atomic_long_read(&lock->owner); + for (;;) { + MUTEX_WARN_ON(__owner_task(owner) != current); + MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP); - if (atomic_long_try_cmpxchg_release(&lock->owner, &owner, __owner_flags(owner))) { - if (owner & MUTEX_FLAG_WAITERS) + if (owner & MUTEX_FLAG_HANDOFF) break; - return; + if (atomic_long_try_cmpxchg_release(&lock->owner, &owner, + __owner_flags(owner))) { + if (owner & MUTEX_FLAG_WAITERS) + break; + return; + } } } -- 2.43.0.472.g3155946c3a-goog
| |