Messages in this thread Patch in this message | | | Date | Fri, 10 Dec 2021 17:16:27 +0100 | From | Peter Zijlstra <> | Subject | [PATCH v2 9/9] refcount: Optimize __refcount_add_not_zero(.i=1) |
| |
Allow the same off-by-one on the target range that refcount_inc() already has by only testing the new value for overflow when the increment is not constant-1.
Improves code-gen, for the common case from:
a887: 41 8b 14 24 mov (%r12),%edx a88b: 83 fa ff cmp $0xffffffff,%edx a88e: 74 1f je a8af <ring_buffer_get+0x3f> a890: 8d 4a 01 lea 0x1(%rdx),%ecx a893: 89 d0 mov %edx,%eax a895: f0 41 0f b1 0c 24 lock cmpxchg %ecx,(%r12) a89b: 75 35 jne a8d2 <ring_buffer_get+0x62> a89d: 83 c2 02 add $0x2,%edx a8a0: 09 ca or %ecx,%edx a8a2: 78 19 js a8bd <ring_buffer_get+0x4d>
to:
a887: 41 8b 04 24 mov (%r12),%eax a88b: 83 f8 ff cmp $0xffffffff,%eax a88e: 74 1a je a8aa <ring_buffer_get+0x3a> a890: 8d 50 01 lea 0x1(%rax),%edx a893: f0 41 0f b1 14 24 lock cmpxchg %edx,(%r12) a899: 75 f0 jne a88b <ring_buffer_get+0x1b> a89b: 85 d2 test %edx,%edx a89d: 78 19 js a8b8 <ring_buffer_get+0x48>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> --- include/linux/refcount.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
--- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -161,7 +161,7 @@ static inline __must_check bool __refcou if (oldp) *oldp = old; - if (unlikely(old < 0 || old + i < 0)) + if (unlikely(old < 0 || (!(__builtin_constant_p(i) && i == 1) && old + i < 0))) refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF); return old;
| |