Messages in this thread | | | Date | Sat, 25 Jun 2022 19:15:10 +1000 | Subject | Re: [PATCH] kernfs: Change kernfs_rwsem to a per-cpu rwsem. | From | Imran Khan <> |
| |
Hello Tejun,
On 25/6/22 2:12 pm, Tejun Heo wrote: > On Mon, Jun 20, 2022 at 01:26:34PM +1000, Imran Khan wrote: >> On large systems when few hundred CPUs simulateously acquire kernfs_rwsem, >> for reading we see performance degradation due to bouncing of cache line >> that contains kernfs_rwsem. Changing kernfs_rwsem into a per-fs, per-cpu >> rwsem can fix this degradation. > ... >> Moreover this run of 200 applications take more than 32 secs to finish on >> this system. >> >> After changing kernfs_rwsem to a per-cpu rwsem, I can see that contention >> for kernfs_rwsem is no longer visible in perf data and the test execution >> time has reduced to almost half (17 secs or less from 32 secs or more). >> >> The overhead involving write operations with per-cpu rwsem will be higher >> but frequency of creation/deletion of kernfs files is much less than >> frequency at which kernfs (cgroup, sysfs) files are read. > > The problem with percpu_rwsem is that write locking requires going > through a RCU grace period, which can easily add two or more digit > millisec latencies. I'm pretty sure there are code paths which are > pretty heavy on write - e.g. during boot, depending on the machine > configuration, we could be write-acquiring the rwsem hundreds of > thousands of times and we'd be constantly doing RCU grace periods. > > So, I don't think kernfs_rwsem is a good candidate for percpu rwsem. > There likely are plenty of cases where write path isn't cold enough. >
Thanks for your feedback. I will continue with approach of hashed kernfs_rwsem then.
Thanks -- Imran
| |