Messages in this thread Patch in this message | | | Date | Sat, 19 Aug 2023 06:08:50 +0000 | Subject | [PATCH v5 16/19] sched: Fix proxy/current (push,pull)ability | From | John Stultz <> |
| |
From: Valentin Schneider <valentin.schneider@arm.com>
Proxy execution forms atomic pairs of tasks: The selected task (scheduling context) and an proxy (execution context). The selected task, along with the rest of the blocked chain, follows the proxy wrt CPU placement.
They can be the same task, in which case push/pull doesn't need any modification. When they are different, however, FIFO1 & FIFO42:
,-> RT42 | | blocked-on | v blocked_donor | mutex | | owner | v `-- RT1
RT1 RT42
CPU0 CPU1 ^ ^ | | overloaded !overloaded rq prio = 42 rq prio = 0
RT1 is eligible to be pushed to CPU1, but should that happen it will "carry" RT42 along. Clearly here neither RT1 nor RT42 must be seen as push/pullable.
Unfortunately, only the selected task is usually dequeued from the rq, and the proxy'ed execution context (rq->curr) remains on the rq. This can cause RT1 to be selected for migration from logic like the rt pushable_list.
This patch adds a dequeue/enqueue cycle on the proxy task before __schedule returns, which allows the sched class logic to avoid adding the now current task to the pushable_list.
Furthermore, tasks becoming blocked on a mutex don't need an explicit dequeue/enqueue cycle to be made (push/pull)able: they have to be running to block on a mutex, thus they will eventually hit put_prev_task().
XXX: pinned tasks becoming unblocked should be removed from the push/pull lists, but those don't get to see __schedule() straight away.
Cc: Joel Fernandes <joelaf@google.com> Cc: Qais Yousef <qyousef@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Ben Segall <bsegall@google.com> Cc: Zimuzo Ezeozue <zezeozue@google.com> Cc: Youssef Esmat <youssefesmat@google.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Daniel Bristot de Oliveira <bristot@redhat.com> Cc: Will Deacon <will@kernel.org> Cc: Waiman Long <longman@redhat.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: "Paul E . McKenney" <paulmck@kernel.org> Cc: kernel-team@android.com Signed-off-by: Valentin Schneider <valentin.schneider@arm.com> Signed-off-by: Connor O'Brien <connoro@google.com> Signed-off-by: John Stultz <jstultz@google.com> --- v3: * Tweaked comments & commit message v5: * Minor simplifications to utilize the fix earlier in the patch series. * Rework the wording of the commit message to match selected/proxy terminology and expand a bit to make it more clear how it works. --- kernel/sched/core.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c index bee7082b294f..e8065fc5c894 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6656,6 +6656,21 @@ proxy(struct rq *rq, struct task_struct *next, struct rq_flags *rf) raw_spin_unlock(&mutex->wait_lock); return ret; } + +static inline void proxy_tag_curr(struct rq *rq, struct task_struct *next) +{ + /* + * pick_next_task() calls set_next_task() on the selected task + * at some point, which ensures it is not push/pullable. + * However, the selected task *and* the ,mutex owner form an + * atomic pair wrt push/pull. + * + * Make sure owner is not pushable. Unfortunately we can only + * deal with that by means of a dequeue/enqueue cycle. :-/ + */ + dequeue_task(rq, next, DEQUEUE_NOCLOCK | DEQUEUE_SAVE); + enqueue_task(rq, next, ENQUEUE_NOCLOCK | ENQUEUE_RESTORE); +} #else /* PROXY_EXEC */ static struct task_struct * proxy(struct rq *rq, struct task_struct *next, struct rq_flags *rf) @@ -6663,6 +6678,8 @@ proxy(struct rq *rq, struct task_struct *next, struct rq_flags *rf) BUG(); // This should never be called in the !PROXY case return next; } + +static inline void proxy_tag_curr(struct rq *rq, struct task_struct *next) { } #endif /* PROXY_EXEC */ /* @@ -6711,6 +6728,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) unsigned long prev_state; struct rq_flags rf; struct rq *rq; + bool proxied; int cpu; cpu = smp_processor_id(); @@ -6760,6 +6778,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) switch_count = &prev->nvcsw; } + proxied = (rq_selected(rq) != prev); pick_again: next = pick_next_task(rq, rq_selected(rq), &rf); rq_set_selected(rq, next); @@ -6786,6 +6805,10 @@ static void __sched notrace __schedule(unsigned int sched_mode) * changes to task_struct made by pick_next_task(). */ RCU_INIT_POINTER(rq->curr, next); + + if (unlikely(!task_current_selected(rq, next))) + proxy_tag_curr(rq, next); + /* * The membarrier system call requires each architecture * to have a full memory barrier after updating @@ -6810,6 +6833,10 @@ static void __sched notrace __schedule(unsigned int sched_mode) /* Also unlocks the rq: */ rq = context_switch(rq, prev, next, &rf); } else { + /* In case next was already curr but just got blocked_donor*/ + if (unlikely(!task_current_selected(rq, next))) + proxy_tag_curr(rq, next); + rq->clock_update_flags &= ~(RQCF_ACT_SKIP|RQCF_REQ_SKIP); rq_unpin_lock(rq, &rf); -- 2.42.0.rc1.204.g551eb34607-goog
| |