Messages in this thread Patch in this message | | | From | Waiman Long <> | Subject | [PATCH v2 4/5] locking/rwsem: Wake up all waiting readers if RWSEM_WAKE_READ_OWNED | Date | Fri, 20 Nov 2020 23:14:15 -0500 |
| |
The rwsem wakeup logic has been modified by commit d3681e269fff ("locking/rwsem: Wake up almost all readers in wait queue") to wake up all readers in the wait queue if the first waiter is a reader. This change was made to implement a phase-fair reader/writer lock. Once a reader gets the lock, all the current waiting readers will be allowed to join. Other readers that come after that will not be allowed to prevent writer starvation.
In the case of RWSEM_WAKE_READ_OWNED, not all currently waiting readers can be woken up if the first waiter happens to be a writer. Complete the phase-fair logic by waking up all readers even for this case.
Signed-off-by: Waiman Long <longman@redhat.com> --- kernel/locking/rwsem.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index b373990fcab8..e0ad2019c518 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -404,6 +404,7 @@ static void rwsem_mark_wake(struct rw_semaphore *sem, struct rwsem_waiter *waiter, *tmp; long oldcount, woken = 0, adjustment = 0; struct list_head wlist; + bool first_is_reader = true; lockdep_assert_held(&sem->wait_lock); @@ -426,7 +427,13 @@ static void rwsem_mark_wake(struct rw_semaphore *sem, lockevent_inc(rwsem_wake_writer); } - return; + /* + * If rwsem has already been owned by reader, wake up other + * readers in the wait queue even if first one is a writer. + */ + if (wake_type != RWSEM_WAKE_READ_OWNED) + return; + first_is_reader = false; } /* @@ -520,10 +527,12 @@ static void rwsem_mark_wake(struct rw_semaphore *sem, } /* - * When we've woken a reader, we no longer need to force writers - * to give up the lock and we can clear HANDOFF. + * When readers are woken, we no longer need to force writers to + * give up the lock and we can clear HANDOFF unless the first + * waiter is a writer. */ - if (woken && (atomic_long_read(&sem->count) & RWSEM_FLAG_HANDOFF)) + if (woken && first_is_reader && + (atomic_long_read(&sem->count) & RWSEM_FLAG_HANDOFF)) adjustment -= RWSEM_FLAG_HANDOFF; if (adjustment) @@ -1053,8 +1062,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, int state, long count) if (rwsem_optimistic_spin(sem, false)) { /* rwsem_optimistic_spin() implies ACQUIRE on success */ /* - * Wake up other readers in the wait list if the front - * waiter is a reader. + * Wake up other readers in the wait queue. */ wake_readers: if ((atomic_long_read(&sem->count) & RWSEM_FLAG_WAITERS)) { -- 2.18.1
| |