Messages in this thread | | | Date | Mon, 16 May 2022 09:42:45 +0100 | From | Mark Rutland <> | Subject | Re: [v2 2/2] kbuild: copy scripts/atomic/atomic-*.h to include/generated/atomic-*.h |
| |
[adding the atomics maintainers to Cc]
On Sat, May 14, 2022 at 01:02:07PM +0900, Masahiro Yamada wrote: > include/linux/atomic/*.h are generated by scripts/atomic/gen-atomics.sh. > > To ensure they are not edited, scripts/atomic/check-atomics.sh checks > sha1sum every time. > > This commit moves include/linux/atomic/*.h to scripts/atomic/*.h_shipped, > which are copied to include/generated/ at build time: > > COPY include/generated/atomic-instrumented.h > COPY include/generated/atomic-long.h > COPY include/generated/atomic-arch-fallback.h
FWIW, moving these and copying them at build time is fine by me, given that better indicates that these are generated.
> I dropped the sha1sum checks. I hope nobody would try to directly modify > *_shipped files.
I'd prefer to keep the sha1sum check, because we're worried that people *will* modify them, and that won't be noticed until the next time they're regenerated.
> Kbuild runs more and more tools at build time these days because they > are fast enough on modern systems. > > For example, > > - 29c833061c1d8c2d1d23a62e7061561eadd76cdb > ("kconfig: generate lexer and parser during build instead of shipping") > > - 7c0303ff7e67b637c47d8afee533ca9e2a02359b > ("crypto: arm - generate *.S by Perl at build time instead of shipping them") > > Yet, gen-atomics.sh is too slow.
Yes; we'd originally wanted to run them at build time, but that was too slow, and as a compromise we moved to regenerating them whenever we changed the scripting.
> I guess it can be improved because it is just simple text processing. > Then, Kbuild can execute it at build time.
It is in theory possible to make them much faster, yes. The actual slowness of the scripting is largely due to using sub-shells resulting in a load of fork+exec, which could be avoided if using a real scripting language.
Practically speaking though, we're rather limited in what we can rely upon being available. We chose to use POSIX shell as a lowest-common-demoninator, and anything that'd be *nice* to use isn't going to always be available, meaning that even if we make it faster we can't necessarily build it all the time anyway.
Thanks, Mark.
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > Kbuild | 26 +- > include/linux/atomic/atomic-arch-fallback.h | 2391 +---------------- > include/linux/atomic/atomic-instrumented.h | 2050 +------------- > include/linux/atomic/atomic-long.h | 1016 +------ > .../atomic/atomic-arch-fallback.h_shipped | 1 - > .../atomic/atomic-instrumented.h_shipped | 1 - > .../atomic/atomic-long.h_shipped | 1 - > scripts/atomic/check-atomics.sh | 33 - > scripts/atomic/gen-atomics.sh | 14 +- > 9 files changed, 24 insertions(+), 5509 deletions(-) > copy include/linux/atomic/atomic-arch-fallback.h => scripts/atomic/atomic-arch-fallback.h_shipped (99%) > copy include/linux/atomic/atomic-instrumented.h => scripts/atomic/atomic-instrumented.h_shipped (99%) > copy include/linux/atomic/atomic-long.h => scripts/atomic/atomic-long.h_shipped (99%) > delete mode 100755 scripts/atomic/check-atomics.sh > > diff --git a/Kbuild b/Kbuild > index fa441b98c9f6..768921bca727 100644 > --- a/Kbuild > +++ b/Kbuild > @@ -2,13 +2,24 @@ > # > # Kbuild for top-level directory of the kernel > > +#### > +# Copy atomic headers > + > +atomic-files := $(addprefix include/generated/, \ > + atomic-instrumented.h atomic-long.h atomic-arch-fallback.h) > + > +targets := $(atomic-files) > + > +$(atomic-files): include/generated/%.h: $(srctree)/scripts/atomic/%.h_shipped > + $(call cmd,copy) > + > ##### > # Generate bounds.h > > bounds-file := include/generated/bounds.h > > always-y := $(bounds-file) > -targets := kernel/bounds.s > +targets += kernel/bounds.s > > $(bounds-file): kernel/bounds.s FORCE > $(call filechk,offsets,__LINUX_BOUNDS_H__) > @@ -31,7 +42,7 @@ offsets-file := include/generated/asm-offsets.h > always-y += $(offsets-file) > targets += arch/$(SRCARCH)/kernel/asm-offsets.s > > -arch/$(SRCARCH)/kernel/asm-offsets.s: $(timeconst-file) $(bounds-file) > +arch/$(SRCARCH)/kernel/asm-offsets.s: $(timeconst-file) $(bounds-file) $(atomic-files) > > $(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE > $(call filechk,offsets,__ASM_OFFSETS_H__) > @@ -46,14 +57,3 @@ quiet_cmd_syscalls = CALL $< > > missing-syscalls: scripts/checksyscalls.sh $(offsets-file) FORCE > $(call cmd,syscalls) > - > -##### > -# Check atomic headers are up-to-date > - > -always-y += old-atomics > - > -quiet_cmd_atomics = CALL $< > - cmd_atomics = $(CONFIG_SHELL) $< > - > -old-atomics: scripts/atomic/check-atomics.sh FORCE > - $(call cmd,atomics) > diff --git a/include/linux/atomic/atomic-arch-fallback.h b/include/linux/atomic/atomic-arch-fallback.h > index 6db58d180866..b13148ad6bd6 100644 > --- a/include/linux/atomic/atomic-arch-fallback.h > +++ b/include/linux/atomic/atomic-arch-fallback.h > @@ -1,2389 +1,2 @@ > -// SPDX-License-Identifier: GPL-2.0 > - > -// Generated by scripts/atomic/gen-atomic-fallback.sh > -// DO NOT MODIFY THIS FILE DIRECTLY > - > -#ifndef _LINUX_ATOMIC_FALLBACK_H > -#define _LINUX_ATOMIC_FALLBACK_H > - > -#include <linux/compiler.h> > - > -#ifndef arch_xchg_relaxed > -#define arch_xchg_acquire arch_xchg > -#define arch_xchg_release arch_xchg > -#define arch_xchg_relaxed arch_xchg > -#else /* arch_xchg_relaxed */ > - > -#ifndef arch_xchg_acquire > -#define arch_xchg_acquire(...) \ > - __atomic_op_acquire(arch_xchg, __VA_ARGS__) > -#endif > - > -#ifndef arch_xchg_release > -#define arch_xchg_release(...) \ > - __atomic_op_release(arch_xchg, __VA_ARGS__) > -#endif > - > -#ifndef arch_xchg > -#define arch_xchg(...) \ > - __atomic_op_fence(arch_xchg, __VA_ARGS__) > -#endif > - > -#endif /* arch_xchg_relaxed */ > - > -#ifndef arch_cmpxchg_relaxed > -#define arch_cmpxchg_acquire arch_cmpxchg > -#define arch_cmpxchg_release arch_cmpxchg > -#define arch_cmpxchg_relaxed arch_cmpxchg > -#else /* arch_cmpxchg_relaxed */ > - > -#ifndef arch_cmpxchg_acquire > -#define arch_cmpxchg_acquire(...) \ > - __atomic_op_acquire(arch_cmpxchg, __VA_ARGS__) > -#endif > - > -#ifndef arch_cmpxchg_release > -#define arch_cmpxchg_release(...) \ > - __atomic_op_release(arch_cmpxchg, __VA_ARGS__) > -#endif > - > -#ifndef arch_cmpxchg > -#define arch_cmpxchg(...) \ > - __atomic_op_fence(arch_cmpxchg, __VA_ARGS__) > -#endif > - > -#endif /* arch_cmpxchg_relaxed */ > - > -#ifndef arch_cmpxchg64_relaxed > -#define arch_cmpxchg64_acquire arch_cmpxchg64 > -#define arch_cmpxchg64_release arch_cmpxchg64 > -#define arch_cmpxchg64_relaxed arch_cmpxchg64 > -#else /* arch_cmpxchg64_relaxed */ > - > -#ifndef arch_cmpxchg64_acquire > -#define arch_cmpxchg64_acquire(...) \ > - __atomic_op_acquire(arch_cmpxchg64, __VA_ARGS__) > -#endif > - > -#ifndef arch_cmpxchg64_release > -#define arch_cmpxchg64_release(...) \ > - __atomic_op_release(arch_cmpxchg64, __VA_ARGS__) > -#endif > - > -#ifndef arch_cmpxchg64 > -#define arch_cmpxchg64(...) \ > - __atomic_op_fence(arch_cmpxchg64, __VA_ARGS__) > -#endif > - > -#endif /* arch_cmpxchg64_relaxed */ > - > -#ifndef arch_try_cmpxchg_relaxed > -#ifdef arch_try_cmpxchg > -#define arch_try_cmpxchg_acquire arch_try_cmpxchg > -#define arch_try_cmpxchg_release arch_try_cmpxchg > -#define arch_try_cmpxchg_relaxed arch_try_cmpxchg > -#endif /* arch_try_cmpxchg */ > - > -#ifndef arch_try_cmpxchg > -#define arch_try_cmpxchg(_ptr, _oldp, _new) \ > -({ \ > - typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > - ___r = arch_cmpxchg((_ptr), ___o, (_new)); \ > - if (unlikely(___r != ___o)) \ > - *___op = ___r; \ > - likely(___r == ___o); \ > -}) > -#endif /* arch_try_cmpxchg */ > - > -#ifndef arch_try_cmpxchg_acquire > -#define arch_try_cmpxchg_acquire(_ptr, _oldp, _new) \ > -({ \ > - typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > - ___r = arch_cmpxchg_acquire((_ptr), ___o, (_new)); \ > - if (unlikely(___r != ___o)) \ > - *___op = ___r; \ > - likely(___r == ___o); \ > -}) > -#endif /* arch_try_cmpxchg_acquire */ > - > -#ifndef arch_try_cmpxchg_release > -#define arch_try_cmpxchg_release(_ptr, _oldp, _new) \ > -({ \ > - typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > - ___r = arch_cmpxchg_release((_ptr), ___o, (_new)); \ > - if (unlikely(___r != ___o)) \ > - *___op = ___r; \ > - likely(___r == ___o); \ > -}) > -#endif /* arch_try_cmpxchg_release */ > - > -#ifndef arch_try_cmpxchg_relaxed > -#define arch_try_cmpxchg_relaxed(_ptr, _oldp, _new) \ > -({ \ > - typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > - ___r = arch_cmpxchg_relaxed((_ptr), ___o, (_new)); \ > - if (unlikely(___r != ___o)) \ > - *___op = ___r; \ > - likely(___r == ___o); \ > -}) > -#endif /* arch_try_cmpxchg_relaxed */ > - > -#else /* arch_try_cmpxchg_relaxed */ > - > -#ifndef arch_try_cmpxchg_acquire > -#define arch_try_cmpxchg_acquire(...) \ > - __atomic_op_acquire(arch_try_cmpxchg, __VA_ARGS__) > -#endif > - > -#ifndef arch_try_cmpxchg_release > -#define arch_try_cmpxchg_release(...) \ > - __atomic_op_release(arch_try_cmpxchg, __VA_ARGS__) > -#endif > - > -#ifndef arch_try_cmpxchg > -#define arch_try_cmpxchg(...) \ > - __atomic_op_fence(arch_try_cmpxchg, __VA_ARGS__) > -#endif > - > -#endif /* arch_try_cmpxchg_relaxed */ > - > -#ifndef arch_atomic_read_acquire > -static __always_inline int > -arch_atomic_read_acquire(const atomic_t *v) > -{ > - int ret; > - > - if (__native_word(atomic_t)) { > - ret = smp_load_acquire(&(v)->counter); > - } else { > - ret = arch_atomic_read(v); > - __atomic_acquire_fence(); > - } > - > - return ret; > -} > -#define arch_atomic_read_acquire arch_atomic_read_acquire > -#endif > - > -#ifndef arch_atomic_set_release > -static __always_inline void > -arch_atomic_set_release(atomic_t *v, int i) > -{ > - if (__native_word(atomic_t)) { > - smp_store_release(&(v)->counter, i); > - } else { > - __atomic_release_fence(); > - arch_atomic_set(v, i); > - } > -} > -#define arch_atomic_set_release arch_atomic_set_release > -#endif > - > -#ifndef arch_atomic_add_return_relaxed > -#define arch_atomic_add_return_acquire arch_atomic_add_return > -#define arch_atomic_add_return_release arch_atomic_add_return > -#define arch_atomic_add_return_relaxed arch_atomic_add_return > -#else /* arch_atomic_add_return_relaxed */ > - > -#ifndef arch_atomic_add_return_acquire > -static __always_inline int > -arch_atomic_add_return_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_add_return_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_add_return_acquire arch_atomic_add_return_acquire > -#endif > - > -#ifndef arch_atomic_add_return_release > -static __always_inline int > -arch_atomic_add_return_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_add_return_relaxed(i, v); > -} > -#define arch_atomic_add_return_release arch_atomic_add_return_release > -#endif > - > -#ifndef arch_atomic_add_return > -static __always_inline int > -arch_atomic_add_return(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_add_return_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_add_return arch_atomic_add_return > -#endif > - > -#endif /* arch_atomic_add_return_relaxed */ > - > -#ifndef arch_atomic_fetch_add_relaxed > -#define arch_atomic_fetch_add_acquire arch_atomic_fetch_add > -#define arch_atomic_fetch_add_release arch_atomic_fetch_add > -#define arch_atomic_fetch_add_relaxed arch_atomic_fetch_add > -#else /* arch_atomic_fetch_add_relaxed */ > - > -#ifndef arch_atomic_fetch_add_acquire > -static __always_inline int > -arch_atomic_fetch_add_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_fetch_add_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_add_acquire arch_atomic_fetch_add_acquire > -#endif > - > -#ifndef arch_atomic_fetch_add_release > -static __always_inline int > -arch_atomic_fetch_add_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_add_relaxed(i, v); > -} > -#define arch_atomic_fetch_add_release arch_atomic_fetch_add_release > -#endif > - > -#ifndef arch_atomic_fetch_add > -static __always_inline int > -arch_atomic_fetch_add(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_add_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_add arch_atomic_fetch_add > -#endif > - > -#endif /* arch_atomic_fetch_add_relaxed */ > - > -#ifndef arch_atomic_sub_return_relaxed > -#define arch_atomic_sub_return_acquire arch_atomic_sub_return > -#define arch_atomic_sub_return_release arch_atomic_sub_return > -#define arch_atomic_sub_return_relaxed arch_atomic_sub_return > -#else /* arch_atomic_sub_return_relaxed */ > - > -#ifndef arch_atomic_sub_return_acquire > -static __always_inline int > -arch_atomic_sub_return_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_sub_return_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_sub_return_acquire arch_atomic_sub_return_acquire > -#endif > - > -#ifndef arch_atomic_sub_return_release > -static __always_inline int > -arch_atomic_sub_return_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_sub_return_relaxed(i, v); > -} > -#define arch_atomic_sub_return_release arch_atomic_sub_return_release > -#endif > - > -#ifndef arch_atomic_sub_return > -static __always_inline int > -arch_atomic_sub_return(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_sub_return_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_sub_return arch_atomic_sub_return > -#endif > - > -#endif /* arch_atomic_sub_return_relaxed */ > - > -#ifndef arch_atomic_fetch_sub_relaxed > -#define arch_atomic_fetch_sub_acquire arch_atomic_fetch_sub > -#define arch_atomic_fetch_sub_release arch_atomic_fetch_sub > -#define arch_atomic_fetch_sub_relaxed arch_atomic_fetch_sub > -#else /* arch_atomic_fetch_sub_relaxed */ > - > -#ifndef arch_atomic_fetch_sub_acquire > -static __always_inline int > -arch_atomic_fetch_sub_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_fetch_sub_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_sub_acquire arch_atomic_fetch_sub_acquire > -#endif > - > -#ifndef arch_atomic_fetch_sub_release > -static __always_inline int > -arch_atomic_fetch_sub_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_sub_relaxed(i, v); > -} > -#define arch_atomic_fetch_sub_release arch_atomic_fetch_sub_release > -#endif > - > -#ifndef arch_atomic_fetch_sub > -static __always_inline int > -arch_atomic_fetch_sub(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_sub_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_sub arch_atomic_fetch_sub > -#endif > - > -#endif /* arch_atomic_fetch_sub_relaxed */ > - > -#ifndef arch_atomic_inc > -static __always_inline void > -arch_atomic_inc(atomic_t *v) > -{ > - arch_atomic_add(1, v); > -} > -#define arch_atomic_inc arch_atomic_inc > -#endif > - > -#ifndef arch_atomic_inc_return_relaxed > -#ifdef arch_atomic_inc_return > -#define arch_atomic_inc_return_acquire arch_atomic_inc_return > -#define arch_atomic_inc_return_release arch_atomic_inc_return > -#define arch_atomic_inc_return_relaxed arch_atomic_inc_return > -#endif /* arch_atomic_inc_return */ > - > -#ifndef arch_atomic_inc_return > -static __always_inline int > -arch_atomic_inc_return(atomic_t *v) > -{ > - return arch_atomic_add_return(1, v); > -} > -#define arch_atomic_inc_return arch_atomic_inc_return > -#endif > - > -#ifndef arch_atomic_inc_return_acquire > -static __always_inline int > -arch_atomic_inc_return_acquire(atomic_t *v) > -{ > - return arch_atomic_add_return_acquire(1, v); > -} > -#define arch_atomic_inc_return_acquire arch_atomic_inc_return_acquire > -#endif > - > -#ifndef arch_atomic_inc_return_release > -static __always_inline int > -arch_atomic_inc_return_release(atomic_t *v) > -{ > - return arch_atomic_add_return_release(1, v); > -} > -#define arch_atomic_inc_return_release arch_atomic_inc_return_release > -#endif > - > -#ifndef arch_atomic_inc_return_relaxed > -static __always_inline int > -arch_atomic_inc_return_relaxed(atomic_t *v) > -{ > - return arch_atomic_add_return_relaxed(1, v); > -} > -#define arch_atomic_inc_return_relaxed arch_atomic_inc_return_relaxed > -#endif > - > -#else /* arch_atomic_inc_return_relaxed */ > - > -#ifndef arch_atomic_inc_return_acquire > -static __always_inline int > -arch_atomic_inc_return_acquire(atomic_t *v) > -{ > - int ret = arch_atomic_inc_return_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_inc_return_acquire arch_atomic_inc_return_acquire > -#endif > - > -#ifndef arch_atomic_inc_return_release > -static __always_inline int > -arch_atomic_inc_return_release(atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_inc_return_relaxed(v); > -} > -#define arch_atomic_inc_return_release arch_atomic_inc_return_release > -#endif > - > -#ifndef arch_atomic_inc_return > -static __always_inline int > -arch_atomic_inc_return(atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_inc_return_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_inc_return arch_atomic_inc_return > -#endif > - > -#endif /* arch_atomic_inc_return_relaxed */ > - > -#ifndef arch_atomic_fetch_inc_relaxed > -#ifdef arch_atomic_fetch_inc > -#define arch_atomic_fetch_inc_acquire arch_atomic_fetch_inc > -#define arch_atomic_fetch_inc_release arch_atomic_fetch_inc > -#define arch_atomic_fetch_inc_relaxed arch_atomic_fetch_inc > -#endif /* arch_atomic_fetch_inc */ > - > -#ifndef arch_atomic_fetch_inc > -static __always_inline int > -arch_atomic_fetch_inc(atomic_t *v) > -{ > - return arch_atomic_fetch_add(1, v); > -} > -#define arch_atomic_fetch_inc arch_atomic_fetch_inc > -#endif > - > -#ifndef arch_atomic_fetch_inc_acquire > -static __always_inline int > -arch_atomic_fetch_inc_acquire(atomic_t *v) > -{ > - return arch_atomic_fetch_add_acquire(1, v); > -} > -#define arch_atomic_fetch_inc_acquire arch_atomic_fetch_inc_acquire > -#endif > - > -#ifndef arch_atomic_fetch_inc_release > -static __always_inline int > -arch_atomic_fetch_inc_release(atomic_t *v) > -{ > - return arch_atomic_fetch_add_release(1, v); > -} > -#define arch_atomic_fetch_inc_release arch_atomic_fetch_inc_release > -#endif > - > -#ifndef arch_atomic_fetch_inc_relaxed > -static __always_inline int > -arch_atomic_fetch_inc_relaxed(atomic_t *v) > -{ > - return arch_atomic_fetch_add_relaxed(1, v); > -} > -#define arch_atomic_fetch_inc_relaxed arch_atomic_fetch_inc_relaxed > -#endif > - > -#else /* arch_atomic_fetch_inc_relaxed */ > - > -#ifndef arch_atomic_fetch_inc_acquire > -static __always_inline int > -arch_atomic_fetch_inc_acquire(atomic_t *v) > -{ > - int ret = arch_atomic_fetch_inc_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_inc_acquire arch_atomic_fetch_inc_acquire > -#endif > - > -#ifndef arch_atomic_fetch_inc_release > -static __always_inline int > -arch_atomic_fetch_inc_release(atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_inc_relaxed(v); > -} > -#define arch_atomic_fetch_inc_release arch_atomic_fetch_inc_release > -#endif > - > -#ifndef arch_atomic_fetch_inc > -static __always_inline int > -arch_atomic_fetch_inc(atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_inc_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_inc arch_atomic_fetch_inc > -#endif > - > -#endif /* arch_atomic_fetch_inc_relaxed */ > - > -#ifndef arch_atomic_dec > -static __always_inline void > -arch_atomic_dec(atomic_t *v) > -{ > - arch_atomic_sub(1, v); > -} > -#define arch_atomic_dec arch_atomic_dec > -#endif > - > -#ifndef arch_atomic_dec_return_relaxed > -#ifdef arch_atomic_dec_return > -#define arch_atomic_dec_return_acquire arch_atomic_dec_return > -#define arch_atomic_dec_return_release arch_atomic_dec_return > -#define arch_atomic_dec_return_relaxed arch_atomic_dec_return > -#endif /* arch_atomic_dec_return */ > - > -#ifndef arch_atomic_dec_return > -static __always_inline int > -arch_atomic_dec_return(atomic_t *v) > -{ > - return arch_atomic_sub_return(1, v); > -} > -#define arch_atomic_dec_return arch_atomic_dec_return > -#endif > - > -#ifndef arch_atomic_dec_return_acquire > -static __always_inline int > -arch_atomic_dec_return_acquire(atomic_t *v) > -{ > - return arch_atomic_sub_return_acquire(1, v); > -} > -#define arch_atomic_dec_return_acquire arch_atomic_dec_return_acquire > -#endif > - > -#ifndef arch_atomic_dec_return_release > -static __always_inline int > -arch_atomic_dec_return_release(atomic_t *v) > -{ > - return arch_atomic_sub_return_release(1, v); > -} > -#define arch_atomic_dec_return_release arch_atomic_dec_return_release > -#endif > - > -#ifndef arch_atomic_dec_return_relaxed > -static __always_inline int > -arch_atomic_dec_return_relaxed(atomic_t *v) > -{ > - return arch_atomic_sub_return_relaxed(1, v); > -} > -#define arch_atomic_dec_return_relaxed arch_atomic_dec_return_relaxed > -#endif > - > -#else /* arch_atomic_dec_return_relaxed */ > - > -#ifndef arch_atomic_dec_return_acquire > -static __always_inline int > -arch_atomic_dec_return_acquire(atomic_t *v) > -{ > - int ret = arch_atomic_dec_return_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_dec_return_acquire arch_atomic_dec_return_acquire > -#endif > - > -#ifndef arch_atomic_dec_return_release > -static __always_inline int > -arch_atomic_dec_return_release(atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_dec_return_relaxed(v); > -} > -#define arch_atomic_dec_return_release arch_atomic_dec_return_release > -#endif > - > -#ifndef arch_atomic_dec_return > -static __always_inline int > -arch_atomic_dec_return(atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_dec_return_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_dec_return arch_atomic_dec_return > -#endif > - > -#endif /* arch_atomic_dec_return_relaxed */ > - > -#ifndef arch_atomic_fetch_dec_relaxed > -#ifdef arch_atomic_fetch_dec > -#define arch_atomic_fetch_dec_acquire arch_atomic_fetch_dec > -#define arch_atomic_fetch_dec_release arch_atomic_fetch_dec > -#define arch_atomic_fetch_dec_relaxed arch_atomic_fetch_dec > -#endif /* arch_atomic_fetch_dec */ > - > -#ifndef arch_atomic_fetch_dec > -static __always_inline int > -arch_atomic_fetch_dec(atomic_t *v) > -{ > - return arch_atomic_fetch_sub(1, v); > -} > -#define arch_atomic_fetch_dec arch_atomic_fetch_dec > -#endif > - > -#ifndef arch_atomic_fetch_dec_acquire > -static __always_inline int > -arch_atomic_fetch_dec_acquire(atomic_t *v) > -{ > - return arch_atomic_fetch_sub_acquire(1, v); > -} > -#define arch_atomic_fetch_dec_acquire arch_atomic_fetch_dec_acquire > -#endif > - > -#ifndef arch_atomic_fetch_dec_release > -static __always_inline int > -arch_atomic_fetch_dec_release(atomic_t *v) > -{ > - return arch_atomic_fetch_sub_release(1, v); > -} > -#define arch_atomic_fetch_dec_release arch_atomic_fetch_dec_release > -#endif > - > -#ifndef arch_atomic_fetch_dec_relaxed > -static __always_inline int > -arch_atomic_fetch_dec_relaxed(atomic_t *v) > -{ > - return arch_atomic_fetch_sub_relaxed(1, v); > -} > -#define arch_atomic_fetch_dec_relaxed arch_atomic_fetch_dec_relaxed > -#endif > - > -#else /* arch_atomic_fetch_dec_relaxed */ > - > -#ifndef arch_atomic_fetch_dec_acquire > -static __always_inline int > -arch_atomic_fetch_dec_acquire(atomic_t *v) > -{ > - int ret = arch_atomic_fetch_dec_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_dec_acquire arch_atomic_fetch_dec_acquire > -#endif > - > -#ifndef arch_atomic_fetch_dec_release > -static __always_inline int > -arch_atomic_fetch_dec_release(atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_dec_relaxed(v); > -} > -#define arch_atomic_fetch_dec_release arch_atomic_fetch_dec_release > -#endif > - > -#ifndef arch_atomic_fetch_dec > -static __always_inline int > -arch_atomic_fetch_dec(atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_dec_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_dec arch_atomic_fetch_dec > -#endif > - > -#endif /* arch_atomic_fetch_dec_relaxed */ > - > -#ifndef arch_atomic_fetch_and_relaxed > -#define arch_atomic_fetch_and_acquire arch_atomic_fetch_and > -#define arch_atomic_fetch_and_release arch_atomic_fetch_and > -#define arch_atomic_fetch_and_relaxed arch_atomic_fetch_and > -#else /* arch_atomic_fetch_and_relaxed */ > - > -#ifndef arch_atomic_fetch_and_acquire > -static __always_inline int > -arch_atomic_fetch_and_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_fetch_and_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_and_acquire arch_atomic_fetch_and_acquire > -#endif > - > -#ifndef arch_atomic_fetch_and_release > -static __always_inline int > -arch_atomic_fetch_and_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_and_relaxed(i, v); > -} > -#define arch_atomic_fetch_and_release arch_atomic_fetch_and_release > -#endif > - > -#ifndef arch_atomic_fetch_and > -static __always_inline int > -arch_atomic_fetch_and(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_and_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_and arch_atomic_fetch_and > -#endif > - > -#endif /* arch_atomic_fetch_and_relaxed */ > - > -#ifndef arch_atomic_andnot > -static __always_inline void > -arch_atomic_andnot(int i, atomic_t *v) > -{ > - arch_atomic_and(~i, v); > -} > -#define arch_atomic_andnot arch_atomic_andnot > -#endif > - > -#ifndef arch_atomic_fetch_andnot_relaxed > -#ifdef arch_atomic_fetch_andnot > -#define arch_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot > -#define arch_atomic_fetch_andnot_release arch_atomic_fetch_andnot > -#define arch_atomic_fetch_andnot_relaxed arch_atomic_fetch_andnot > -#endif /* arch_atomic_fetch_andnot */ > - > -#ifndef arch_atomic_fetch_andnot > -static __always_inline int > -arch_atomic_fetch_andnot(int i, atomic_t *v) > -{ > - return arch_atomic_fetch_and(~i, v); > -} > -#define arch_atomic_fetch_andnot arch_atomic_fetch_andnot > -#endif > - > -#ifndef arch_atomic_fetch_andnot_acquire > -static __always_inline int > -arch_atomic_fetch_andnot_acquire(int i, atomic_t *v) > -{ > - return arch_atomic_fetch_and_acquire(~i, v); > -} > -#define arch_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot_acquire > -#endif > - > -#ifndef arch_atomic_fetch_andnot_release > -static __always_inline int > -arch_atomic_fetch_andnot_release(int i, atomic_t *v) > -{ > - return arch_atomic_fetch_and_release(~i, v); > -} > -#define arch_atomic_fetch_andnot_release arch_atomic_fetch_andnot_release > -#endif > - > -#ifndef arch_atomic_fetch_andnot_relaxed > -static __always_inline int > -arch_atomic_fetch_andnot_relaxed(int i, atomic_t *v) > -{ > - return arch_atomic_fetch_and_relaxed(~i, v); > -} > -#define arch_atomic_fetch_andnot_relaxed arch_atomic_fetch_andnot_relaxed > -#endif > - > -#else /* arch_atomic_fetch_andnot_relaxed */ > - > -#ifndef arch_atomic_fetch_andnot_acquire > -static __always_inline int > -arch_atomic_fetch_andnot_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_fetch_andnot_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot_acquire > -#endif > - > -#ifndef arch_atomic_fetch_andnot_release > -static __always_inline int > -arch_atomic_fetch_andnot_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_andnot_relaxed(i, v); > -} > -#define arch_atomic_fetch_andnot_release arch_atomic_fetch_andnot_release > -#endif > - > -#ifndef arch_atomic_fetch_andnot > -static __always_inline int > -arch_atomic_fetch_andnot(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_andnot_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_andnot arch_atomic_fetch_andnot > -#endif > - > -#endif /* arch_atomic_fetch_andnot_relaxed */ > - > -#ifndef arch_atomic_fetch_or_relaxed > -#define arch_atomic_fetch_or_acquire arch_atomic_fetch_or > -#define arch_atomic_fetch_or_release arch_atomic_fetch_or > -#define arch_atomic_fetch_or_relaxed arch_atomic_fetch_or > -#else /* arch_atomic_fetch_or_relaxed */ > - > -#ifndef arch_atomic_fetch_or_acquire > -static __always_inline int > -arch_atomic_fetch_or_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_fetch_or_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_or_acquire arch_atomic_fetch_or_acquire > -#endif > - > -#ifndef arch_atomic_fetch_or_release > -static __always_inline int > -arch_atomic_fetch_or_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_or_relaxed(i, v); > -} > -#define arch_atomic_fetch_or_release arch_atomic_fetch_or_release > -#endif > - > -#ifndef arch_atomic_fetch_or > -static __always_inline int > -arch_atomic_fetch_or(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_or_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_or arch_atomic_fetch_or > -#endif > - > -#endif /* arch_atomic_fetch_or_relaxed */ > - > -#ifndef arch_atomic_fetch_xor_relaxed > -#define arch_atomic_fetch_xor_acquire arch_atomic_fetch_xor > -#define arch_atomic_fetch_xor_release arch_atomic_fetch_xor > -#define arch_atomic_fetch_xor_relaxed arch_atomic_fetch_xor > -#else /* arch_atomic_fetch_xor_relaxed */ > - > -#ifndef arch_atomic_fetch_xor_acquire > -static __always_inline int > -arch_atomic_fetch_xor_acquire(int i, atomic_t *v) > -{ > - int ret = arch_atomic_fetch_xor_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_fetch_xor_acquire arch_atomic_fetch_xor_acquire > -#endif > - > -#ifndef arch_atomic_fetch_xor_release > -static __always_inline int > -arch_atomic_fetch_xor_release(int i, atomic_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic_fetch_xor_relaxed(i, v); > -} > -#define arch_atomic_fetch_xor_release arch_atomic_fetch_xor_release > -#endif > - > -#ifndef arch_atomic_fetch_xor > -static __always_inline int > -arch_atomic_fetch_xor(int i, atomic_t *v) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_fetch_xor_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_fetch_xor arch_atomic_fetch_xor > -#endif > - > -#endif /* arch_atomic_fetch_xor_relaxed */ > - > -#ifndef arch_atomic_xchg_relaxed > -#define arch_atomic_xchg_acquire arch_atomic_xchg > -#define arch_atomic_xchg_release arch_atomic_xchg > -#define arch_atomic_xchg_relaxed arch_atomic_xchg > -#else /* arch_atomic_xchg_relaxed */ > - > -#ifndef arch_atomic_xchg_acquire > -static __always_inline int > -arch_atomic_xchg_acquire(atomic_t *v, int i) > -{ > - int ret = arch_atomic_xchg_relaxed(v, i); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_xchg_acquire arch_atomic_xchg_acquire > -#endif > - > -#ifndef arch_atomic_xchg_release > -static __always_inline int > -arch_atomic_xchg_release(atomic_t *v, int i) > -{ > - __atomic_release_fence(); > - return arch_atomic_xchg_relaxed(v, i); > -} > -#define arch_atomic_xchg_release arch_atomic_xchg_release > -#endif > - > -#ifndef arch_atomic_xchg > -static __always_inline int > -arch_atomic_xchg(atomic_t *v, int i) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_xchg_relaxed(v, i); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_xchg arch_atomic_xchg > -#endif > - > -#endif /* arch_atomic_xchg_relaxed */ > - > -#ifndef arch_atomic_cmpxchg_relaxed > -#define arch_atomic_cmpxchg_acquire arch_atomic_cmpxchg > -#define arch_atomic_cmpxchg_release arch_atomic_cmpxchg > -#define arch_atomic_cmpxchg_relaxed arch_atomic_cmpxchg > -#else /* arch_atomic_cmpxchg_relaxed */ > - > -#ifndef arch_atomic_cmpxchg_acquire > -static __always_inline int > -arch_atomic_cmpxchg_acquire(atomic_t *v, int old, int new) > -{ > - int ret = arch_atomic_cmpxchg_relaxed(v, old, new); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_cmpxchg_acquire arch_atomic_cmpxchg_acquire > -#endif > - > -#ifndef arch_atomic_cmpxchg_release > -static __always_inline int > -arch_atomic_cmpxchg_release(atomic_t *v, int old, int new) > -{ > - __atomic_release_fence(); > - return arch_atomic_cmpxchg_relaxed(v, old, new); > -} > -#define arch_atomic_cmpxchg_release arch_atomic_cmpxchg_release > -#endif > - > -#ifndef arch_atomic_cmpxchg > -static __always_inline int > -arch_atomic_cmpxchg(atomic_t *v, int old, int new) > -{ > - int ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_cmpxchg_relaxed(v, old, new); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_cmpxchg arch_atomic_cmpxchg > -#endif > - > -#endif /* arch_atomic_cmpxchg_relaxed */ > - > -#ifndef arch_atomic_try_cmpxchg_relaxed > -#ifdef arch_atomic_try_cmpxchg > -#define arch_atomic_try_cmpxchg_acquire arch_atomic_try_cmpxchg > -#define arch_atomic_try_cmpxchg_release arch_atomic_try_cmpxchg > -#define arch_atomic_try_cmpxchg_relaxed arch_atomic_try_cmpxchg > -#endif /* arch_atomic_try_cmpxchg */ > - > -#ifndef arch_atomic_try_cmpxchg > -static __always_inline bool > -arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) > -{ > - int r, o = *old; > - r = arch_atomic_cmpxchg(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg > -#endif > - > -#ifndef arch_atomic_try_cmpxchg_acquire > -static __always_inline bool > -arch_atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new) > -{ > - int r, o = *old; > - r = arch_atomic_cmpxchg_acquire(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic_try_cmpxchg_acquire arch_atomic_try_cmpxchg_acquire > -#endif > - > -#ifndef arch_atomic_try_cmpxchg_release > -static __always_inline bool > -arch_atomic_try_cmpxchg_release(atomic_t *v, int *old, int new) > -{ > - int r, o = *old; > - r = arch_atomic_cmpxchg_release(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic_try_cmpxchg_release arch_atomic_try_cmpxchg_release > -#endif > - > -#ifndef arch_atomic_try_cmpxchg_relaxed > -static __always_inline bool > -arch_atomic_try_cmpxchg_relaxed(atomic_t *v, int *old, int new) > -{ > - int r, o = *old; > - r = arch_atomic_cmpxchg_relaxed(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic_try_cmpxchg_relaxed arch_atomic_try_cmpxchg_relaxed > -#endif > - > -#else /* arch_atomic_try_cmpxchg_relaxed */ > - > -#ifndef arch_atomic_try_cmpxchg_acquire > -static __always_inline bool > -arch_atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new) > -{ > - bool ret = arch_atomic_try_cmpxchg_relaxed(v, old, new); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic_try_cmpxchg_acquire arch_atomic_try_cmpxchg_acquire > -#endif > - > -#ifndef arch_atomic_try_cmpxchg_release > -static __always_inline bool > -arch_atomic_try_cmpxchg_release(atomic_t *v, int *old, int new) > -{ > - __atomic_release_fence(); > - return arch_atomic_try_cmpxchg_relaxed(v, old, new); > -} > -#define arch_atomic_try_cmpxchg_release arch_atomic_try_cmpxchg_release > -#endif > - > -#ifndef arch_atomic_try_cmpxchg > -static __always_inline bool > -arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) > -{ > - bool ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic_try_cmpxchg_relaxed(v, old, new); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg > -#endif > - > -#endif /* arch_atomic_try_cmpxchg_relaxed */ > - > -#ifndef arch_atomic_sub_and_test > -/** > - * arch_atomic_sub_and_test - subtract value from variable and test result > - * @i: integer value to subtract > - * @v: pointer of type atomic_t > - * > - * Atomically subtracts @i from @v and returns > - * true if the result is zero, or false for all > - * other cases. > - */ > -static __always_inline bool > -arch_atomic_sub_and_test(int i, atomic_t *v) > -{ > - return arch_atomic_sub_return(i, v) == 0; > -} > -#define arch_atomic_sub_and_test arch_atomic_sub_and_test > -#endif > - > -#ifndef arch_atomic_dec_and_test > -/** > - * arch_atomic_dec_and_test - decrement and test > - * @v: pointer of type atomic_t > - * > - * Atomically decrements @v by 1 and > - * returns true if the result is 0, or false for all other > - * cases. > - */ > -static __always_inline bool > -arch_atomic_dec_and_test(atomic_t *v) > -{ > - return arch_atomic_dec_return(v) == 0; > -} > -#define arch_atomic_dec_and_test arch_atomic_dec_and_test > -#endif > - > -#ifndef arch_atomic_inc_and_test > -/** > - * arch_atomic_inc_and_test - increment and test > - * @v: pointer of type atomic_t > - * > - * Atomically increments @v by 1 > - * and returns true if the result is zero, or false for all > - * other cases. > - */ > -static __always_inline bool > -arch_atomic_inc_and_test(atomic_t *v) > -{ > - return arch_atomic_inc_return(v) == 0; > -} > -#define arch_atomic_inc_and_test arch_atomic_inc_and_test > -#endif > - > -#ifndef arch_atomic_add_negative > -/** > - * arch_atomic_add_negative - add and test if negative > - * @i: integer value to add > - * @v: pointer of type atomic_t > - * > - * Atomically adds @i to @v and returns true > - * if the result is negative, or false when > - * result is greater than or equal to zero. > - */ > -static __always_inline bool > -arch_atomic_add_negative(int i, atomic_t *v) > -{ > - return arch_atomic_add_return(i, v) < 0; > -} > -#define arch_atomic_add_negative arch_atomic_add_negative > -#endif > - > -#ifndef arch_atomic_fetch_add_unless > -/** > - * arch_atomic_fetch_add_unless - add unless the number is already a given value > - * @v: pointer of type atomic_t > - * @a: the amount to add to v... > - * @u: ...unless v is equal to u. > - * > - * Atomically adds @a to @v, so long as @v was not already @u. > - * Returns original value of @v > - */ > -static __always_inline int > -arch_atomic_fetch_add_unless(atomic_t *v, int a, int u) > -{ > - int c = arch_atomic_read(v); > - > - do { > - if (unlikely(c == u)) > - break; > - } while (!arch_atomic_try_cmpxchg(v, &c, c + a)); > - > - return c; > -} > -#define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless > -#endif > - > -#ifndef arch_atomic_add_unless > -/** > - * arch_atomic_add_unless - add unless the number is already a given value > - * @v: pointer of type atomic_t > - * @a: the amount to add to v... > - * @u: ...unless v is equal to u. > - * > - * Atomically adds @a to @v, if @v was not already @u. > - * Returns true if the addition was done. > - */ > -static __always_inline bool > -arch_atomic_add_unless(atomic_t *v, int a, int u) > -{ > - return arch_atomic_fetch_add_unless(v, a, u) != u; > -} > -#define arch_atomic_add_unless arch_atomic_add_unless > -#endif > - > -#ifndef arch_atomic_inc_not_zero > -/** > - * arch_atomic_inc_not_zero - increment unless the number is zero > - * @v: pointer of type atomic_t > - * > - * Atomically increments @v by 1, if @v is non-zero. > - * Returns true if the increment was done. > - */ > -static __always_inline bool > -arch_atomic_inc_not_zero(atomic_t *v) > -{ > - return arch_atomic_add_unless(v, 1, 0); > -} > -#define arch_atomic_inc_not_zero arch_atomic_inc_not_zero > -#endif > - > -#ifndef arch_atomic_inc_unless_negative > -static __always_inline bool > -arch_atomic_inc_unless_negative(atomic_t *v) > -{ > - int c = arch_atomic_read(v); > - > - do { > - if (unlikely(c < 0)) > - return false; > - } while (!arch_atomic_try_cmpxchg(v, &c, c + 1)); > - > - return true; > -} > -#define arch_atomic_inc_unless_negative arch_atomic_inc_unless_negative > -#endif > - > -#ifndef arch_atomic_dec_unless_positive > -static __always_inline bool > -arch_atomic_dec_unless_positive(atomic_t *v) > -{ > - int c = arch_atomic_read(v); > - > - do { > - if (unlikely(c > 0)) > - return false; > - } while (!arch_atomic_try_cmpxchg(v, &c, c - 1)); > - > - return true; > -} > -#define arch_atomic_dec_unless_positive arch_atomic_dec_unless_positive > -#endif > - > -#ifndef arch_atomic_dec_if_positive > -static __always_inline int > -arch_atomic_dec_if_positive(atomic_t *v) > -{ > - int dec, c = arch_atomic_read(v); > - > - do { > - dec = c - 1; > - if (unlikely(dec < 0)) > - break; > - } while (!arch_atomic_try_cmpxchg(v, &c, dec)); > - > - return dec; > -} > -#define arch_atomic_dec_if_positive arch_atomic_dec_if_positive > -#endif > - > -#ifdef CONFIG_GENERIC_ATOMIC64 > -#include <asm-generic/atomic64.h> > -#endif > - > -#ifndef arch_atomic64_read_acquire > -static __always_inline s64 > -arch_atomic64_read_acquire(const atomic64_t *v) > -{ > - s64 ret; > - > - if (__native_word(atomic64_t)) { > - ret = smp_load_acquire(&(v)->counter); > - } else { > - ret = arch_atomic64_read(v); > - __atomic_acquire_fence(); > - } > - > - return ret; > -} > -#define arch_atomic64_read_acquire arch_atomic64_read_acquire > -#endif > - > -#ifndef arch_atomic64_set_release > -static __always_inline void > -arch_atomic64_set_release(atomic64_t *v, s64 i) > -{ > - if (__native_word(atomic64_t)) { > - smp_store_release(&(v)->counter, i); > - } else { > - __atomic_release_fence(); > - arch_atomic64_set(v, i); > - } > -} > -#define arch_atomic64_set_release arch_atomic64_set_release > -#endif > - > -#ifndef arch_atomic64_add_return_relaxed > -#define arch_atomic64_add_return_acquire arch_atomic64_add_return > -#define arch_atomic64_add_return_release arch_atomic64_add_return > -#define arch_atomic64_add_return_relaxed arch_atomic64_add_return > -#else /* arch_atomic64_add_return_relaxed */ > - > -#ifndef arch_atomic64_add_return_acquire > -static __always_inline s64 > -arch_atomic64_add_return_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_add_return_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_add_return_acquire arch_atomic64_add_return_acquire > -#endif > - > -#ifndef arch_atomic64_add_return_release > -static __always_inline s64 > -arch_atomic64_add_return_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_add_return_relaxed(i, v); > -} > -#define arch_atomic64_add_return_release arch_atomic64_add_return_release > -#endif > - > -#ifndef arch_atomic64_add_return > -static __always_inline s64 > -arch_atomic64_add_return(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_add_return_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_add_return arch_atomic64_add_return > -#endif > - > -#endif /* arch_atomic64_add_return_relaxed */ > - > -#ifndef arch_atomic64_fetch_add_relaxed > -#define arch_atomic64_fetch_add_acquire arch_atomic64_fetch_add > -#define arch_atomic64_fetch_add_release arch_atomic64_fetch_add > -#define arch_atomic64_fetch_add_relaxed arch_atomic64_fetch_add > -#else /* arch_atomic64_fetch_add_relaxed */ > - > -#ifndef arch_atomic64_fetch_add_acquire > -static __always_inline s64 > -arch_atomic64_fetch_add_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_add_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_add_acquire arch_atomic64_fetch_add_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_add_release > -static __always_inline s64 > -arch_atomic64_fetch_add_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_add_relaxed(i, v); > -} > -#define arch_atomic64_fetch_add_release arch_atomic64_fetch_add_release > -#endif > - > -#ifndef arch_atomic64_fetch_add > -static __always_inline s64 > -arch_atomic64_fetch_add(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_add_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_add arch_atomic64_fetch_add > -#endif > - > -#endif /* arch_atomic64_fetch_add_relaxed */ > - > -#ifndef arch_atomic64_sub_return_relaxed > -#define arch_atomic64_sub_return_acquire arch_atomic64_sub_return > -#define arch_atomic64_sub_return_release arch_atomic64_sub_return > -#define arch_atomic64_sub_return_relaxed arch_atomic64_sub_return > -#else /* arch_atomic64_sub_return_relaxed */ > - > -#ifndef arch_atomic64_sub_return_acquire > -static __always_inline s64 > -arch_atomic64_sub_return_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_sub_return_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_sub_return_acquire arch_atomic64_sub_return_acquire > -#endif > - > -#ifndef arch_atomic64_sub_return_release > -static __always_inline s64 > -arch_atomic64_sub_return_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_sub_return_relaxed(i, v); > -} > -#define arch_atomic64_sub_return_release arch_atomic64_sub_return_release > -#endif > - > -#ifndef arch_atomic64_sub_return > -static __always_inline s64 > -arch_atomic64_sub_return(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_sub_return_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_sub_return arch_atomic64_sub_return > -#endif > - > -#endif /* arch_atomic64_sub_return_relaxed */ > - > -#ifndef arch_atomic64_fetch_sub_relaxed > -#define arch_atomic64_fetch_sub_acquire arch_atomic64_fetch_sub > -#define arch_atomic64_fetch_sub_release arch_atomic64_fetch_sub > -#define arch_atomic64_fetch_sub_relaxed arch_atomic64_fetch_sub > -#else /* arch_atomic64_fetch_sub_relaxed */ > - > -#ifndef arch_atomic64_fetch_sub_acquire > -static __always_inline s64 > -arch_atomic64_fetch_sub_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_sub_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_sub_acquire arch_atomic64_fetch_sub_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_sub_release > -static __always_inline s64 > -arch_atomic64_fetch_sub_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_sub_relaxed(i, v); > -} > -#define arch_atomic64_fetch_sub_release arch_atomic64_fetch_sub_release > -#endif > - > -#ifndef arch_atomic64_fetch_sub > -static __always_inline s64 > -arch_atomic64_fetch_sub(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_sub_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_sub arch_atomic64_fetch_sub > -#endif > - > -#endif /* arch_atomic64_fetch_sub_relaxed */ > - > -#ifndef arch_atomic64_inc > -static __always_inline void > -arch_atomic64_inc(atomic64_t *v) > -{ > - arch_atomic64_add(1, v); > -} > -#define arch_atomic64_inc arch_atomic64_inc > -#endif > - > -#ifndef arch_atomic64_inc_return_relaxed > -#ifdef arch_atomic64_inc_return > -#define arch_atomic64_inc_return_acquire arch_atomic64_inc_return > -#define arch_atomic64_inc_return_release arch_atomic64_inc_return > -#define arch_atomic64_inc_return_relaxed arch_atomic64_inc_return > -#endif /* arch_atomic64_inc_return */ > - > -#ifndef arch_atomic64_inc_return > -static __always_inline s64 > -arch_atomic64_inc_return(atomic64_t *v) > -{ > - return arch_atomic64_add_return(1, v); > -} > -#define arch_atomic64_inc_return arch_atomic64_inc_return > -#endif > - > -#ifndef arch_atomic64_inc_return_acquire > -static __always_inline s64 > -arch_atomic64_inc_return_acquire(atomic64_t *v) > -{ > - return arch_atomic64_add_return_acquire(1, v); > -} > -#define arch_atomic64_inc_return_acquire arch_atomic64_inc_return_acquire > -#endif > - > -#ifndef arch_atomic64_inc_return_release > -static __always_inline s64 > -arch_atomic64_inc_return_release(atomic64_t *v) > -{ > - return arch_atomic64_add_return_release(1, v); > -} > -#define arch_atomic64_inc_return_release arch_atomic64_inc_return_release > -#endif > - > -#ifndef arch_atomic64_inc_return_relaxed > -static __always_inline s64 > -arch_atomic64_inc_return_relaxed(atomic64_t *v) > -{ > - return arch_atomic64_add_return_relaxed(1, v); > -} > -#define arch_atomic64_inc_return_relaxed arch_atomic64_inc_return_relaxed > -#endif > - > -#else /* arch_atomic64_inc_return_relaxed */ > - > -#ifndef arch_atomic64_inc_return_acquire > -static __always_inline s64 > -arch_atomic64_inc_return_acquire(atomic64_t *v) > -{ > - s64 ret = arch_atomic64_inc_return_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_inc_return_acquire arch_atomic64_inc_return_acquire > -#endif > - > -#ifndef arch_atomic64_inc_return_release > -static __always_inline s64 > -arch_atomic64_inc_return_release(atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_inc_return_relaxed(v); > -} > -#define arch_atomic64_inc_return_release arch_atomic64_inc_return_release > -#endif > - > -#ifndef arch_atomic64_inc_return > -static __always_inline s64 > -arch_atomic64_inc_return(atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_inc_return_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_inc_return arch_atomic64_inc_return > -#endif > - > -#endif /* arch_atomic64_inc_return_relaxed */ > - > -#ifndef arch_atomic64_fetch_inc_relaxed > -#ifdef arch_atomic64_fetch_inc > -#define arch_atomic64_fetch_inc_acquire arch_atomic64_fetch_inc > -#define arch_atomic64_fetch_inc_release arch_atomic64_fetch_inc > -#define arch_atomic64_fetch_inc_relaxed arch_atomic64_fetch_inc > -#endif /* arch_atomic64_fetch_inc */ > - > -#ifndef arch_atomic64_fetch_inc > -static __always_inline s64 > -arch_atomic64_fetch_inc(atomic64_t *v) > -{ > - return arch_atomic64_fetch_add(1, v); > -} > -#define arch_atomic64_fetch_inc arch_atomic64_fetch_inc > -#endif > - > -#ifndef arch_atomic64_fetch_inc_acquire > -static __always_inline s64 > -arch_atomic64_fetch_inc_acquire(atomic64_t *v) > -{ > - return arch_atomic64_fetch_add_acquire(1, v); > -} > -#define arch_atomic64_fetch_inc_acquire arch_atomic64_fetch_inc_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_inc_release > -static __always_inline s64 > -arch_atomic64_fetch_inc_release(atomic64_t *v) > -{ > - return arch_atomic64_fetch_add_release(1, v); > -} > -#define arch_atomic64_fetch_inc_release arch_atomic64_fetch_inc_release > -#endif > - > -#ifndef arch_atomic64_fetch_inc_relaxed > -static __always_inline s64 > -arch_atomic64_fetch_inc_relaxed(atomic64_t *v) > -{ > - return arch_atomic64_fetch_add_relaxed(1, v); > -} > -#define arch_atomic64_fetch_inc_relaxed arch_atomic64_fetch_inc_relaxed > -#endif > - > -#else /* arch_atomic64_fetch_inc_relaxed */ > - > -#ifndef arch_atomic64_fetch_inc_acquire > -static __always_inline s64 > -arch_atomic64_fetch_inc_acquire(atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_inc_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_inc_acquire arch_atomic64_fetch_inc_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_inc_release > -static __always_inline s64 > -arch_atomic64_fetch_inc_release(atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_inc_relaxed(v); > -} > -#define arch_atomic64_fetch_inc_release arch_atomic64_fetch_inc_release > -#endif > - > -#ifndef arch_atomic64_fetch_inc > -static __always_inline s64 > -arch_atomic64_fetch_inc(atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_inc_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_inc arch_atomic64_fetch_inc > -#endif > - > -#endif /* arch_atomic64_fetch_inc_relaxed */ > - > -#ifndef arch_atomic64_dec > -static __always_inline void > -arch_atomic64_dec(atomic64_t *v) > -{ > - arch_atomic64_sub(1, v); > -} > -#define arch_atomic64_dec arch_atomic64_dec > -#endif > - > -#ifndef arch_atomic64_dec_return_relaxed > -#ifdef arch_atomic64_dec_return > -#define arch_atomic64_dec_return_acquire arch_atomic64_dec_return > -#define arch_atomic64_dec_return_release arch_atomic64_dec_return > -#define arch_atomic64_dec_return_relaxed arch_atomic64_dec_return > -#endif /* arch_atomic64_dec_return */ > - > -#ifndef arch_atomic64_dec_return > -static __always_inline s64 > -arch_atomic64_dec_return(atomic64_t *v) > -{ > - return arch_atomic64_sub_return(1, v); > -} > -#define arch_atomic64_dec_return arch_atomic64_dec_return > -#endif > - > -#ifndef arch_atomic64_dec_return_acquire > -static __always_inline s64 > -arch_atomic64_dec_return_acquire(atomic64_t *v) > -{ > - return arch_atomic64_sub_return_acquire(1, v); > -} > -#define arch_atomic64_dec_return_acquire arch_atomic64_dec_return_acquire > -#endif > - > -#ifndef arch_atomic64_dec_return_release > -static __always_inline s64 > -arch_atomic64_dec_return_release(atomic64_t *v) > -{ > - return arch_atomic64_sub_return_release(1, v); > -} > -#define arch_atomic64_dec_return_release arch_atomic64_dec_return_release > -#endif > - > -#ifndef arch_atomic64_dec_return_relaxed > -static __always_inline s64 > -arch_atomic64_dec_return_relaxed(atomic64_t *v) > -{ > - return arch_atomic64_sub_return_relaxed(1, v); > -} > -#define arch_atomic64_dec_return_relaxed arch_atomic64_dec_return_relaxed > -#endif > - > -#else /* arch_atomic64_dec_return_relaxed */ > - > -#ifndef arch_atomic64_dec_return_acquire > -static __always_inline s64 > -arch_atomic64_dec_return_acquire(atomic64_t *v) > -{ > - s64 ret = arch_atomic64_dec_return_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_dec_return_acquire arch_atomic64_dec_return_acquire > -#endif > - > -#ifndef arch_atomic64_dec_return_release > -static __always_inline s64 > -arch_atomic64_dec_return_release(atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_dec_return_relaxed(v); > -} > -#define arch_atomic64_dec_return_release arch_atomic64_dec_return_release > -#endif > - > -#ifndef arch_atomic64_dec_return > -static __always_inline s64 > -arch_atomic64_dec_return(atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_dec_return_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_dec_return arch_atomic64_dec_return > -#endif > - > -#endif /* arch_atomic64_dec_return_relaxed */ > - > -#ifndef arch_atomic64_fetch_dec_relaxed > -#ifdef arch_atomic64_fetch_dec > -#define arch_atomic64_fetch_dec_acquire arch_atomic64_fetch_dec > -#define arch_atomic64_fetch_dec_release arch_atomic64_fetch_dec > -#define arch_atomic64_fetch_dec_relaxed arch_atomic64_fetch_dec > -#endif /* arch_atomic64_fetch_dec */ > - > -#ifndef arch_atomic64_fetch_dec > -static __always_inline s64 > -arch_atomic64_fetch_dec(atomic64_t *v) > -{ > - return arch_atomic64_fetch_sub(1, v); > -} > -#define arch_atomic64_fetch_dec arch_atomic64_fetch_dec > -#endif > - > -#ifndef arch_atomic64_fetch_dec_acquire > -static __always_inline s64 > -arch_atomic64_fetch_dec_acquire(atomic64_t *v) > -{ > - return arch_atomic64_fetch_sub_acquire(1, v); > -} > -#define arch_atomic64_fetch_dec_acquire arch_atomic64_fetch_dec_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_dec_release > -static __always_inline s64 > -arch_atomic64_fetch_dec_release(atomic64_t *v) > -{ > - return arch_atomic64_fetch_sub_release(1, v); > -} > -#define arch_atomic64_fetch_dec_release arch_atomic64_fetch_dec_release > -#endif > - > -#ifndef arch_atomic64_fetch_dec_relaxed > -static __always_inline s64 > -arch_atomic64_fetch_dec_relaxed(atomic64_t *v) > -{ > - return arch_atomic64_fetch_sub_relaxed(1, v); > -} > -#define arch_atomic64_fetch_dec_relaxed arch_atomic64_fetch_dec_relaxed > -#endif > - > -#else /* arch_atomic64_fetch_dec_relaxed */ > - > -#ifndef arch_atomic64_fetch_dec_acquire > -static __always_inline s64 > -arch_atomic64_fetch_dec_acquire(atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_dec_relaxed(v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_dec_acquire arch_atomic64_fetch_dec_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_dec_release > -static __always_inline s64 > -arch_atomic64_fetch_dec_release(atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_dec_relaxed(v); > -} > -#define arch_atomic64_fetch_dec_release arch_atomic64_fetch_dec_release > -#endif > - > -#ifndef arch_atomic64_fetch_dec > -static __always_inline s64 > -arch_atomic64_fetch_dec(atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_dec_relaxed(v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_dec arch_atomic64_fetch_dec > -#endif > - > -#endif /* arch_atomic64_fetch_dec_relaxed */ > - > -#ifndef arch_atomic64_fetch_and_relaxed > -#define arch_atomic64_fetch_and_acquire arch_atomic64_fetch_and > -#define arch_atomic64_fetch_and_release arch_atomic64_fetch_and > -#define arch_atomic64_fetch_and_relaxed arch_atomic64_fetch_and > -#else /* arch_atomic64_fetch_and_relaxed */ > - > -#ifndef arch_atomic64_fetch_and_acquire > -static __always_inline s64 > -arch_atomic64_fetch_and_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_and_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_and_acquire arch_atomic64_fetch_and_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_and_release > -static __always_inline s64 > -arch_atomic64_fetch_and_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_and_relaxed(i, v); > -} > -#define arch_atomic64_fetch_and_release arch_atomic64_fetch_and_release > -#endif > - > -#ifndef arch_atomic64_fetch_and > -static __always_inline s64 > -arch_atomic64_fetch_and(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_and_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_and arch_atomic64_fetch_and > -#endif > - > -#endif /* arch_atomic64_fetch_and_relaxed */ > - > -#ifndef arch_atomic64_andnot > -static __always_inline void > -arch_atomic64_andnot(s64 i, atomic64_t *v) > -{ > - arch_atomic64_and(~i, v); > -} > -#define arch_atomic64_andnot arch_atomic64_andnot > -#endif > - > -#ifndef arch_atomic64_fetch_andnot_relaxed > -#ifdef arch_atomic64_fetch_andnot > -#define arch_atomic64_fetch_andnot_acquire arch_atomic64_fetch_andnot > -#define arch_atomic64_fetch_andnot_release arch_atomic64_fetch_andnot > -#define arch_atomic64_fetch_andnot_relaxed arch_atomic64_fetch_andnot > -#endif /* arch_atomic64_fetch_andnot */ > - > -#ifndef arch_atomic64_fetch_andnot > -static __always_inline s64 > -arch_atomic64_fetch_andnot(s64 i, atomic64_t *v) > -{ > - return arch_atomic64_fetch_and(~i, v); > -} > -#define arch_atomic64_fetch_andnot arch_atomic64_fetch_andnot > -#endif > - > -#ifndef arch_atomic64_fetch_andnot_acquire > -static __always_inline s64 > -arch_atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v) > -{ > - return arch_atomic64_fetch_and_acquire(~i, v); > -} > -#define arch_atomic64_fetch_andnot_acquire arch_atomic64_fetch_andnot_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_andnot_release > -static __always_inline s64 > -arch_atomic64_fetch_andnot_release(s64 i, atomic64_t *v) > -{ > - return arch_atomic64_fetch_and_release(~i, v); > -} > -#define arch_atomic64_fetch_andnot_release arch_atomic64_fetch_andnot_release > -#endif > - > -#ifndef arch_atomic64_fetch_andnot_relaxed > -static __always_inline s64 > -arch_atomic64_fetch_andnot_relaxed(s64 i, atomic64_t *v) > -{ > - return arch_atomic64_fetch_and_relaxed(~i, v); > -} > -#define arch_atomic64_fetch_andnot_relaxed arch_atomic64_fetch_andnot_relaxed > -#endif > - > -#else /* arch_atomic64_fetch_andnot_relaxed */ > - > -#ifndef arch_atomic64_fetch_andnot_acquire > -static __always_inline s64 > -arch_atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_andnot_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_andnot_acquire arch_atomic64_fetch_andnot_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_andnot_release > -static __always_inline s64 > -arch_atomic64_fetch_andnot_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_andnot_relaxed(i, v); > -} > -#define arch_atomic64_fetch_andnot_release arch_atomic64_fetch_andnot_release > -#endif > - > -#ifndef arch_atomic64_fetch_andnot > -static __always_inline s64 > -arch_atomic64_fetch_andnot(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_andnot_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_andnot arch_atomic64_fetch_andnot > -#endif > - > -#endif /* arch_atomic64_fetch_andnot_relaxed */ > - > -#ifndef arch_atomic64_fetch_or_relaxed > -#define arch_atomic64_fetch_or_acquire arch_atomic64_fetch_or > -#define arch_atomic64_fetch_or_release arch_atomic64_fetch_or > -#define arch_atomic64_fetch_or_relaxed arch_atomic64_fetch_or > -#else /* arch_atomic64_fetch_or_relaxed */ > - > -#ifndef arch_atomic64_fetch_or_acquire > -static __always_inline s64 > -arch_atomic64_fetch_or_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_or_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_or_acquire arch_atomic64_fetch_or_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_or_release > -static __always_inline s64 > -arch_atomic64_fetch_or_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_or_relaxed(i, v); > -} > -#define arch_atomic64_fetch_or_release arch_atomic64_fetch_or_release > -#endif > - > -#ifndef arch_atomic64_fetch_or > -static __always_inline s64 > -arch_atomic64_fetch_or(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_or_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_or arch_atomic64_fetch_or > -#endif > - > -#endif /* arch_atomic64_fetch_or_relaxed */ > - > -#ifndef arch_atomic64_fetch_xor_relaxed > -#define arch_atomic64_fetch_xor_acquire arch_atomic64_fetch_xor > -#define arch_atomic64_fetch_xor_release arch_atomic64_fetch_xor > -#define arch_atomic64_fetch_xor_relaxed arch_atomic64_fetch_xor > -#else /* arch_atomic64_fetch_xor_relaxed */ > - > -#ifndef arch_atomic64_fetch_xor_acquire > -static __always_inline s64 > -arch_atomic64_fetch_xor_acquire(s64 i, atomic64_t *v) > -{ > - s64 ret = arch_atomic64_fetch_xor_relaxed(i, v); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_xor_acquire arch_atomic64_fetch_xor_acquire > -#endif > - > -#ifndef arch_atomic64_fetch_xor_release > -static __always_inline s64 > -arch_atomic64_fetch_xor_release(s64 i, atomic64_t *v) > -{ > - __atomic_release_fence(); > - return arch_atomic64_fetch_xor_relaxed(i, v); > -} > -#define arch_atomic64_fetch_xor_release arch_atomic64_fetch_xor_release > -#endif > - > -#ifndef arch_atomic64_fetch_xor > -static __always_inline s64 > -arch_atomic64_fetch_xor(s64 i, atomic64_t *v) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_fetch_xor_relaxed(i, v); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_fetch_xor arch_atomic64_fetch_xor > -#endif > - > -#endif /* arch_atomic64_fetch_xor_relaxed */ > - > -#ifndef arch_atomic64_xchg_relaxed > -#define arch_atomic64_xchg_acquire arch_atomic64_xchg > -#define arch_atomic64_xchg_release arch_atomic64_xchg > -#define arch_atomic64_xchg_relaxed arch_atomic64_xchg > -#else /* arch_atomic64_xchg_relaxed */ > - > -#ifndef arch_atomic64_xchg_acquire > -static __always_inline s64 > -arch_atomic64_xchg_acquire(atomic64_t *v, s64 i) > -{ > - s64 ret = arch_atomic64_xchg_relaxed(v, i); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_xchg_acquire arch_atomic64_xchg_acquire > -#endif > - > -#ifndef arch_atomic64_xchg_release > -static __always_inline s64 > -arch_atomic64_xchg_release(atomic64_t *v, s64 i) > -{ > - __atomic_release_fence(); > - return arch_atomic64_xchg_relaxed(v, i); > -} > -#define arch_atomic64_xchg_release arch_atomic64_xchg_release > -#endif > - > -#ifndef arch_atomic64_xchg > -static __always_inline s64 > -arch_atomic64_xchg(atomic64_t *v, s64 i) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_xchg_relaxed(v, i); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_xchg arch_atomic64_xchg > -#endif > - > -#endif /* arch_atomic64_xchg_relaxed */ > - > -#ifndef arch_atomic64_cmpxchg_relaxed > -#define arch_atomic64_cmpxchg_acquire arch_atomic64_cmpxchg > -#define arch_atomic64_cmpxchg_release arch_atomic64_cmpxchg > -#define arch_atomic64_cmpxchg_relaxed arch_atomic64_cmpxchg > -#else /* arch_atomic64_cmpxchg_relaxed */ > - > -#ifndef arch_atomic64_cmpxchg_acquire > -static __always_inline s64 > -arch_atomic64_cmpxchg_acquire(atomic64_t *v, s64 old, s64 new) > -{ > - s64 ret = arch_atomic64_cmpxchg_relaxed(v, old, new); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_cmpxchg_acquire arch_atomic64_cmpxchg_acquire > -#endif > - > -#ifndef arch_atomic64_cmpxchg_release > -static __always_inline s64 > -arch_atomic64_cmpxchg_release(atomic64_t *v, s64 old, s64 new) > -{ > - __atomic_release_fence(); > - return arch_atomic64_cmpxchg_relaxed(v, old, new); > -} > -#define arch_atomic64_cmpxchg_release arch_atomic64_cmpxchg_release > -#endif > - > -#ifndef arch_atomic64_cmpxchg > -static __always_inline s64 > -arch_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) > -{ > - s64 ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_cmpxchg_relaxed(v, old, new); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_cmpxchg arch_atomic64_cmpxchg > -#endif > - > -#endif /* arch_atomic64_cmpxchg_relaxed */ > - > -#ifndef arch_atomic64_try_cmpxchg_relaxed > -#ifdef arch_atomic64_try_cmpxchg > -#define arch_atomic64_try_cmpxchg_acquire arch_atomic64_try_cmpxchg > -#define arch_atomic64_try_cmpxchg_release arch_atomic64_try_cmpxchg > -#define arch_atomic64_try_cmpxchg_relaxed arch_atomic64_try_cmpxchg > -#endif /* arch_atomic64_try_cmpxchg */ > - > -#ifndef arch_atomic64_try_cmpxchg > -static __always_inline bool > -arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new) > -{ > - s64 r, o = *old; > - r = arch_atomic64_cmpxchg(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg > -#endif > - > -#ifndef arch_atomic64_try_cmpxchg_acquire > -static __always_inline bool > -arch_atomic64_try_cmpxchg_acquire(atomic64_t *v, s64 *old, s64 new) > -{ > - s64 r, o = *old; > - r = arch_atomic64_cmpxchg_acquire(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic64_try_cmpxchg_acquire arch_atomic64_try_cmpxchg_acquire > -#endif > - > -#ifndef arch_atomic64_try_cmpxchg_release > -static __always_inline bool > -arch_atomic64_try_cmpxchg_release(atomic64_t *v, s64 *old, s64 new) > -{ > - s64 r, o = *old; > - r = arch_atomic64_cmpxchg_release(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic64_try_cmpxchg_release arch_atomic64_try_cmpxchg_release > -#endif > - > -#ifndef arch_atomic64_try_cmpxchg_relaxed > -static __always_inline bool > -arch_atomic64_try_cmpxchg_relaxed(atomic64_t *v, s64 *old, s64 new) > -{ > - s64 r, o = *old; > - r = arch_atomic64_cmpxchg_relaxed(v, o, new); > - if (unlikely(r != o)) > - *old = r; > - return likely(r == o); > -} > -#define arch_atomic64_try_cmpxchg_relaxed arch_atomic64_try_cmpxchg_relaxed > -#endif > - > -#else /* arch_atomic64_try_cmpxchg_relaxed */ > - > -#ifndef arch_atomic64_try_cmpxchg_acquire > -static __always_inline bool > -arch_atomic64_try_cmpxchg_acquire(atomic64_t *v, s64 *old, s64 new) > -{ > - bool ret = arch_atomic64_try_cmpxchg_relaxed(v, old, new); > - __atomic_acquire_fence(); > - return ret; > -} > -#define arch_atomic64_try_cmpxchg_acquire arch_atomic64_try_cmpxchg_acquire > -#endif > - > -#ifndef arch_atomic64_try_cmpxchg_release > -static __always_inline bool > -arch_atomic64_try_cmpxchg_release(atomic64_t *v, s64 *old, s64 new) > -{ > - __atomic_release_fence(); > - return arch_atomic64_try_cmpxchg_relaxed(v, old, new); > -} > -#define arch_atomic64_try_cmpxchg_release arch_atomic64_try_cmpxchg_release > -#endif > - > -#ifndef arch_atomic64_try_cmpxchg > -static __always_inline bool > -arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new) > -{ > - bool ret; > - __atomic_pre_full_fence(); > - ret = arch_atomic64_try_cmpxchg_relaxed(v, old, new); > - __atomic_post_full_fence(); > - return ret; > -} > -#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg > -#endif > - > -#endif /* arch_atomic64_try_cmpxchg_relaxed */ > - > -#ifndef arch_atomic64_sub_and_test > -/** > - * arch_atomic64_sub_and_test - subtract value from variable and test result > - * @i: integer value to subtract > - * @v: pointer of type atomic64_t > - * > - * Atomically subtracts @i from @v and returns > - * true if the result is zero, or false for all > - * other cases. > - */ > -static __always_inline bool > -arch_atomic64_sub_and_test(s64 i, atomic64_t *v) > -{ > - return arch_atomic64_sub_return(i, v) == 0; > -} > -#define arch_atomic64_sub_and_test arch_atomic64_sub_and_test > -#endif > - > -#ifndef arch_atomic64_dec_and_test > -/** > - * arch_atomic64_dec_and_test - decrement and test > - * @v: pointer of type atomic64_t > - * > - * Atomically decrements @v by 1 and > - * returns true if the result is 0, or false for all other > - * cases. > - */ > -static __always_inline bool > -arch_atomic64_dec_and_test(atomic64_t *v) > -{ > - return arch_atomic64_dec_return(v) == 0; > -} > -#define arch_atomic64_dec_and_test arch_atomic64_dec_and_test > -#endif > - > -#ifndef arch_atomic64_inc_and_test > -/** > - * arch_atomic64_inc_and_test - increment and test > - * @v: pointer of type atomic64_t > - * > - * Atomically increments @v by 1 > - * and returns true if the result is zero, or false for all > - * other cases. > - */ > -static __always_inline bool > -arch_atomic64_inc_and_test(atomic64_t *v) > -{ > - return arch_atomic64_inc_return(v) == 0; > -} > -#define arch_atomic64_inc_and_test arch_atomic64_inc_and_test > -#endif > - > -#ifndef arch_atomic64_add_negative > -/** > - * arch_atomic64_add_negative - add and test if negative > - * @i: integer value to add > - * @v: pointer of type atomic64_t > - * > - * Atomically adds @i to @v and returns true > - * if the result is negative, or false when > - * result is greater than or equal to zero. > - */ > -static __always_inline bool > -arch_atomic64_add_negative(s64 i, atomic64_t *v) > -{ > - return arch_atomic64_add_return(i, v) < 0; > -} > -#define arch_atomic64_add_negative arch_atomic64_add_negative > -#endif > - > -#ifndef arch_atomic64_fetch_add_unless > -/** > - * arch_atomic64_fetch_add_unless - add unless the number is already a given value > - * @v: pointer of type atomic64_t > - * @a: the amount to add to v... > - * @u: ...unless v is equal to u. > - * > - * Atomically adds @a to @v, so long as @v was not already @u. > - * Returns original value of @v > - */ > -static __always_inline s64 > -arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u) > -{ > - s64 c = arch_atomic64_read(v); > - > - do { > - if (unlikely(c == u)) > - break; > - } while (!arch_atomic64_try_cmpxchg(v, &c, c + a)); > - > - return c; > -} > -#define arch_atomic64_fetch_add_unless arch_atomic64_fetch_add_unless > -#endif > - > -#ifndef arch_atomic64_add_unless > -/** > - * arch_atomic64_add_unless - add unless the number is already a given value > - * @v: pointer of type atomic64_t > - * @a: the amount to add to v... > - * @u: ...unless v is equal to u. > - * > - * Atomically adds @a to @v, if @v was not already @u. > - * Returns true if the addition was done. > - */ > -static __always_inline bool > -arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u) > -{ > - return arch_atomic64_fetch_add_unless(v, a, u) != u; > -} > -#define arch_atomic64_add_unless arch_atomic64_add_unless > -#endif > - > -#ifndef arch_atomic64_inc_not_zero > -/** > - * arch_atomic64_inc_not_zero - increment unless the number is zero > - * @v: pointer of type atomic64_t > - * > - * Atomically increments @v by 1, if @v is non-zero. > - * Returns true if the increment was done. > - */ > -static __always_inline bool > -arch_atomic64_inc_not_zero(atomic64_t *v) > -{ > - return arch_atomic64_add_unless(v, 1, 0); > -} > -#define arch_atomic64_inc_not_zero arch_atomic64_inc_not_zero > -#endif > - > -#ifndef arch_atomic64_inc_unless_negative > -static __always_inline bool > -arch_atomic64_inc_unless_negative(atomic64_t *v) > -{ > - s64 c = arch_atomic64_read(v); > - > - do { > - if (unlikely(c < 0)) > - return false; > - } while (!arch_atomic64_try_cmpxchg(v, &c, c + 1)); > - > - return true; > -} > -#define arch_atomic64_inc_unless_negative arch_atomic64_inc_unless_negative > -#endif > - > -#ifndef arch_atomic64_dec_unless_positive > -static __always_inline bool > -arch_atomic64_dec_unless_positive(atomic64_t *v) > -{ > - s64 c = arch_atomic64_read(v); > - > - do { > - if (unlikely(c > 0)) > - return false; > - } while (!arch_atomic64_try_cmpxchg(v, &c, c - 1)); > - > - return true; > -} > -#define arch_atomic64_dec_unless_positive arch_atomic64_dec_unless_positive > -#endif > - > -#ifndef arch_atomic64_dec_if_positive > -static __always_inline s64 > -arch_atomic64_dec_if_positive(atomic64_t *v) > -{ > - s64 dec, c = arch_atomic64_read(v); > - > - do { > - dec = c - 1; > - if (unlikely(dec < 0)) > - break; > - } while (!arch_atomic64_try_cmpxchg(v, &c, dec)); > - > - return dec; > -} > -#define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive > -#endif > - > -#endif /* _LINUX_ATOMIC_FALLBACK_H */ > -// 8e2cc06bc0d2c0967d2f8424762bd48555ee40ae > +/* SPDX-License-Identifier: GPL-2.0-only */ > +#include <generated/atomic-arch-fallback.h> > diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h > index 5d69b143c28e..4751a8e6bc13 100644 > --- a/include/linux/atomic/atomic-instrumented.h > +++ b/include/linux/atomic/atomic-instrumented.h > @@ -1,2048 +1,2 @@ > -// SPDX-License-Identifier: GPL-2.0 > - > -// Generated by scripts/atomic/gen-atomic-instrumented.sh > -// DO NOT MODIFY THIS FILE DIRECTLY > - > -/* > - * This file provides wrappers with KASAN instrumentation for atomic operations. > - * To use this functionality an arch's atomic.h file needs to define all > - * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include > - * this file at the end. This file provides atomic_read() that forwards to > - * arch_atomic_read() for actual atomic operation. > - * Note: if an arch atomic operation is implemented by means of other atomic > - * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use > - * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid > - * double instrumentation. > - */ > -#ifndef _LINUX_ATOMIC_INSTRUMENTED_H > -#define _LINUX_ATOMIC_INSTRUMENTED_H > - > -#include <linux/build_bug.h> > -#include <linux/compiler.h> > -#include <linux/instrumented.h> > - > -static __always_inline int > -atomic_read(const atomic_t *v) > -{ > - instrument_atomic_read(v, sizeof(*v)); > - return arch_atomic_read(v); > -} > - > -static __always_inline int > -atomic_read_acquire(const atomic_t *v) > -{ > - instrument_atomic_read(v, sizeof(*v)); > - return arch_atomic_read_acquire(v); > -} > - > -static __always_inline void > -atomic_set(atomic_t *v, int i) > -{ > - instrument_atomic_write(v, sizeof(*v)); > - arch_atomic_set(v, i); > -} > - > -static __always_inline void > -atomic_set_release(atomic_t *v, int i) > -{ > - kcsan_release(); > - instrument_atomic_write(v, sizeof(*v)); > - arch_atomic_set_release(v, i); > -} > - > -static __always_inline void > -atomic_add(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_add(i, v); > -} > - > -static __always_inline int > -atomic_add_return(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_add_return(i, v); > -} > - > -static __always_inline int > -atomic_add_return_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_add_return_acquire(i, v); > -} > - > -static __always_inline int > -atomic_add_return_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_add_return_release(i, v); > -} > - > -static __always_inline int > -atomic_add_return_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_add_return_relaxed(i, v); > -} > - > -static __always_inline int > -atomic_fetch_add(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_add(i, v); > -} > - > -static __always_inline int > -atomic_fetch_add_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_add_acquire(i, v); > -} > - > -static __always_inline int > -atomic_fetch_add_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_add_release(i, v); > -} > - > -static __always_inline int > -atomic_fetch_add_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_add_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_sub(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_sub(i, v); > -} > - > -static __always_inline int > -atomic_sub_return(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_sub_return(i, v); > -} > - > -static __always_inline int > -atomic_sub_return_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_sub_return_acquire(i, v); > -} > - > -static __always_inline int > -atomic_sub_return_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_sub_return_release(i, v); > -} > - > -static __always_inline int > -atomic_sub_return_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_sub_return_relaxed(i, v); > -} > - > -static __always_inline int > -atomic_fetch_sub(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_sub(i, v); > -} > - > -static __always_inline int > -atomic_fetch_sub_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_sub_acquire(i, v); > -} > - > -static __always_inline int > -atomic_fetch_sub_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_sub_release(i, v); > -} > - > -static __always_inline int > -atomic_fetch_sub_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_sub_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_inc(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_inc(v); > -} > - > -static __always_inline int > -atomic_inc_return(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_return(v); > -} > - > -static __always_inline int > -atomic_inc_return_acquire(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_return_acquire(v); > -} > - > -static __always_inline int > -atomic_inc_return_release(atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_return_release(v); > -} > - > -static __always_inline int > -atomic_inc_return_relaxed(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_return_relaxed(v); > -} > - > -static __always_inline int > -atomic_fetch_inc(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_inc(v); > -} > - > -static __always_inline int > -atomic_fetch_inc_acquire(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_inc_acquire(v); > -} > - > -static __always_inline int > -atomic_fetch_inc_release(atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_inc_release(v); > -} > - > -static __always_inline int > -atomic_fetch_inc_relaxed(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_inc_relaxed(v); > -} > - > -static __always_inline void > -atomic_dec(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_dec(v); > -} > - > -static __always_inline int > -atomic_dec_return(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_return(v); > -} > - > -static __always_inline int > -atomic_dec_return_acquire(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_return_acquire(v); > -} > - > -static __always_inline int > -atomic_dec_return_release(atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_return_release(v); > -} > - > -static __always_inline int > -atomic_dec_return_relaxed(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_return_relaxed(v); > -} > - > -static __always_inline int > -atomic_fetch_dec(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_dec(v); > -} > - > -static __always_inline int > -atomic_fetch_dec_acquire(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_dec_acquire(v); > -} > - > -static __always_inline int > -atomic_fetch_dec_release(atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_dec_release(v); > -} > - > -static __always_inline int > -atomic_fetch_dec_relaxed(atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_dec_relaxed(v); > -} > - > -static __always_inline void > -atomic_and(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_and(i, v); > -} > - > -static __always_inline int > -atomic_fetch_and(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_and(i, v); > -} > - > -static __always_inline int > -atomic_fetch_and_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_and_acquire(i, v); > -} > - > -static __always_inline int > -atomic_fetch_and_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_and_release(i, v); > -} > - > -static __always_inline int > -atomic_fetch_and_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_and_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_andnot(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_andnot(i, v); > -} > - > -static __always_inline int > -atomic_fetch_andnot(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_andnot(i, v); > -} > - > -static __always_inline int > -atomic_fetch_andnot_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_andnot_acquire(i, v); > -} > - > -static __always_inline int > -atomic_fetch_andnot_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_andnot_release(i, v); > -} > - > -static __always_inline int > -atomic_fetch_andnot_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_andnot_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_or(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_or(i, v); > -} > - > -static __always_inline int > -atomic_fetch_or(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_or(i, v); > -} > - > -static __always_inline int > -atomic_fetch_or_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_or_acquire(i, v); > -} > - > -static __always_inline int > -atomic_fetch_or_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_or_release(i, v); > -} > - > -static __always_inline int > -atomic_fetch_or_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_or_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_xor(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_xor(i, v); > -} > - > -static __always_inline int > -atomic_fetch_xor(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_xor(i, v); > -} > - > -static __always_inline int > -atomic_fetch_xor_acquire(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_xor_acquire(i, v); > -} > - > -static __always_inline int > -atomic_fetch_xor_release(int i, atomic_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_xor_release(i, v); > -} > - > -static __always_inline int > -atomic_fetch_xor_relaxed(int i, atomic_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_xor_relaxed(i, v); > -} > - > -static __always_inline int > -atomic_xchg(atomic_t *v, int i) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_xchg(v, i); > -} > - > -static __always_inline int > -atomic_xchg_acquire(atomic_t *v, int i) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_xchg_acquire(v, i); > -} > - > -static __always_inline int > -atomic_xchg_release(atomic_t *v, int i) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_xchg_release(v, i); > -} > - > -static __always_inline int > -atomic_xchg_relaxed(atomic_t *v, int i) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_xchg_relaxed(v, i); > -} > - > -static __always_inline int > -atomic_cmpxchg(atomic_t *v, int old, int new) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_cmpxchg(v, old, new); > -} > - > -static __always_inline int > -atomic_cmpxchg_acquire(atomic_t *v, int old, int new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline int > -atomic_cmpxchg_release(atomic_t *v, int old, int new) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_cmpxchg_release(v, old, new); > -} > - > -static __always_inline int > -atomic_cmpxchg_relaxed(atomic_t *v, int old, int new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -atomic_try_cmpxchg(atomic_t *v, int *old, int new) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_try_cmpxchg(v, old, new); > -} > - > -static __always_inline bool > -atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_try_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline bool > -atomic_try_cmpxchg_release(atomic_t *v, int *old, int new) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_try_cmpxchg_release(v, old, new); > -} > - > -static __always_inline bool > -atomic_try_cmpxchg_relaxed(atomic_t *v, int *old, int new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_try_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -atomic_sub_and_test(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_sub_and_test(i, v); > -} > - > -static __always_inline bool > -atomic_dec_and_test(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_and_test(v); > -} > - > -static __always_inline bool > -atomic_inc_and_test(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_and_test(v); > -} > - > -static __always_inline bool > -atomic_add_negative(int i, atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_add_negative(i, v); > -} > - > -static __always_inline int > -atomic_fetch_add_unless(atomic_t *v, int a, int u) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_fetch_add_unless(v, a, u); > -} > - > -static __always_inline bool > -atomic_add_unless(atomic_t *v, int a, int u) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_add_unless(v, a, u); > -} > - > -static __always_inline bool > -atomic_inc_not_zero(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_not_zero(v); > -} > - > -static __always_inline bool > -atomic_inc_unless_negative(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_inc_unless_negative(v); > -} > - > -static __always_inline bool > -atomic_dec_unless_positive(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_unless_positive(v); > -} > - > -static __always_inline int > -atomic_dec_if_positive(atomic_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_dec_if_positive(v); > -} > - > -static __always_inline s64 > -atomic64_read(const atomic64_t *v) > -{ > - instrument_atomic_read(v, sizeof(*v)); > - return arch_atomic64_read(v); > -} > - > -static __always_inline s64 > -atomic64_read_acquire(const atomic64_t *v) > -{ > - instrument_atomic_read(v, sizeof(*v)); > - return arch_atomic64_read_acquire(v); > -} > - > -static __always_inline void > -atomic64_set(atomic64_t *v, s64 i) > -{ > - instrument_atomic_write(v, sizeof(*v)); > - arch_atomic64_set(v, i); > -} > - > -static __always_inline void > -atomic64_set_release(atomic64_t *v, s64 i) > -{ > - kcsan_release(); > - instrument_atomic_write(v, sizeof(*v)); > - arch_atomic64_set_release(v, i); > -} > - > -static __always_inline void > -atomic64_add(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_add(i, v); > -} > - > -static __always_inline s64 > -atomic64_add_return(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_add_return(i, v); > -} > - > -static __always_inline s64 > -atomic64_add_return_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_add_return_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_add_return_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_add_return_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_add_return_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_add_return_relaxed(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_add(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_add(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_add_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_add_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_add_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_add_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_add_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_add_relaxed(i, v); > -} > - > -static __always_inline void > -atomic64_sub(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_sub(i, v); > -} > - > -static __always_inline s64 > -atomic64_sub_return(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_sub_return(i, v); > -} > - > -static __always_inline s64 > -atomic64_sub_return_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_sub_return_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_sub_return_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_sub_return_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_sub_return_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_sub_return_relaxed(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_sub(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_sub(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_sub_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_sub_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_sub_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_sub_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_sub_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_sub_relaxed(i, v); > -} > - > -static __always_inline void > -atomic64_inc(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_inc(v); > -} > - > -static __always_inline s64 > -atomic64_inc_return(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_return(v); > -} > - > -static __always_inline s64 > -atomic64_inc_return_acquire(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_return_acquire(v); > -} > - > -static __always_inline s64 > -atomic64_inc_return_release(atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_return_release(v); > -} > - > -static __always_inline s64 > -atomic64_inc_return_relaxed(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_return_relaxed(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_inc(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_inc(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_inc_acquire(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_inc_acquire(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_inc_release(atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_inc_release(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_inc_relaxed(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_inc_relaxed(v); > -} > - > -static __always_inline void > -atomic64_dec(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_dec(v); > -} > - > -static __always_inline s64 > -atomic64_dec_return(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_return(v); > -} > - > -static __always_inline s64 > -atomic64_dec_return_acquire(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_return_acquire(v); > -} > - > -static __always_inline s64 > -atomic64_dec_return_release(atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_return_release(v); > -} > - > -static __always_inline s64 > -atomic64_dec_return_relaxed(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_return_relaxed(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_dec(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_dec(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_dec_acquire(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_dec_acquire(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_dec_release(atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_dec_release(v); > -} > - > -static __always_inline s64 > -atomic64_fetch_dec_relaxed(atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_dec_relaxed(v); > -} > - > -static __always_inline void > -atomic64_and(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_and(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_and(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_and(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_and_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_and_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_and_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_and_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_and_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_and_relaxed(i, v); > -} > - > -static __always_inline void > -atomic64_andnot(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_andnot(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_andnot(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_andnot(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_andnot_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_andnot_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_andnot_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_andnot_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_andnot_relaxed(i, v); > -} > - > -static __always_inline void > -atomic64_or(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_or(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_or(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_or(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_or_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_or_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_or_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_or_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_or_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_or_relaxed(i, v); > -} > - > -static __always_inline void > -atomic64_xor(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic64_xor(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_xor(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_xor(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_xor_acquire(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_xor_acquire(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_xor_release(s64 i, atomic64_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_xor_release(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_xor_relaxed(s64 i, atomic64_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_xor_relaxed(i, v); > -} > - > -static __always_inline s64 > -atomic64_xchg(atomic64_t *v, s64 i) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_xchg(v, i); > -} > - > -static __always_inline s64 > -atomic64_xchg_acquire(atomic64_t *v, s64 i) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_xchg_acquire(v, i); > -} > - > -static __always_inline s64 > -atomic64_xchg_release(atomic64_t *v, s64 i) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_xchg_release(v, i); > -} > - > -static __always_inline s64 > -atomic64_xchg_relaxed(atomic64_t *v, s64 i) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_xchg_relaxed(v, i); > -} > - > -static __always_inline s64 > -atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_cmpxchg(v, old, new); > -} > - > -static __always_inline s64 > -atomic64_cmpxchg_acquire(atomic64_t *v, s64 old, s64 new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline s64 > -atomic64_cmpxchg_release(atomic64_t *v, s64 old, s64 new) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_cmpxchg_release(v, old, new); > -} > - > -static __always_inline s64 > -atomic64_cmpxchg_relaxed(atomic64_t *v, s64 old, s64 new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic64_try_cmpxchg(v, old, new); > -} > - > -static __always_inline bool > -atomic64_try_cmpxchg_acquire(atomic64_t *v, s64 *old, s64 new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic64_try_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline bool > -atomic64_try_cmpxchg_release(atomic64_t *v, s64 *old, s64 new) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic64_try_cmpxchg_release(v, old, new); > -} > - > -static __always_inline bool > -atomic64_try_cmpxchg_relaxed(atomic64_t *v, s64 *old, s64 new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic64_try_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -atomic64_sub_and_test(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_sub_and_test(i, v); > -} > - > -static __always_inline bool > -atomic64_dec_and_test(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_and_test(v); > -} > - > -static __always_inline bool > -atomic64_inc_and_test(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_and_test(v); > -} > - > -static __always_inline bool > -atomic64_add_negative(s64 i, atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_add_negative(i, v); > -} > - > -static __always_inline s64 > -atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_fetch_add_unless(v, a, u); > -} > - > -static __always_inline bool > -atomic64_add_unless(atomic64_t *v, s64 a, s64 u) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_add_unless(v, a, u); > -} > - > -static __always_inline bool > -atomic64_inc_not_zero(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_not_zero(v); > -} > - > -static __always_inline bool > -atomic64_inc_unless_negative(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_inc_unless_negative(v); > -} > - > -static __always_inline bool > -atomic64_dec_unless_positive(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_unless_positive(v); > -} > - > -static __always_inline s64 > -atomic64_dec_if_positive(atomic64_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic64_dec_if_positive(v); > -} > - > -static __always_inline long > -atomic_long_read(const atomic_long_t *v) > -{ > - instrument_atomic_read(v, sizeof(*v)); > - return arch_atomic_long_read(v); > -} > - > -static __always_inline long > -atomic_long_read_acquire(const atomic_long_t *v) > -{ > - instrument_atomic_read(v, sizeof(*v)); > - return arch_atomic_long_read_acquire(v); > -} > - > -static __always_inline void > -atomic_long_set(atomic_long_t *v, long i) > -{ > - instrument_atomic_write(v, sizeof(*v)); > - arch_atomic_long_set(v, i); > -} > - > -static __always_inline void > -atomic_long_set_release(atomic_long_t *v, long i) > -{ > - kcsan_release(); > - instrument_atomic_write(v, sizeof(*v)); > - arch_atomic_long_set_release(v, i); > -} > - > -static __always_inline void > -atomic_long_add(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_add(i, v); > -} > - > -static __always_inline long > -atomic_long_add_return(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_add_return(i, v); > -} > - > -static __always_inline long > -atomic_long_add_return_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_add_return_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_add_return_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_add_return_release(i, v); > -} > - > -static __always_inline long > -atomic_long_add_return_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_add_return_relaxed(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_add(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_add(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_add_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_add_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_add_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_add_release(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_add_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_add_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_long_sub(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_sub(i, v); > -} > - > -static __always_inline long > -atomic_long_sub_return(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_sub_return(i, v); > -} > - > -static __always_inline long > -atomic_long_sub_return_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_sub_return_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_sub_return_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_sub_return_release(i, v); > -} > - > -static __always_inline long > -atomic_long_sub_return_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_sub_return_relaxed(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_sub(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_sub(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_sub_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_sub_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_sub_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_sub_release(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_sub_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_long_inc(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_inc(v); > -} > - > -static __always_inline long > -atomic_long_inc_return(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_return(v); > -} > - > -static __always_inline long > -atomic_long_inc_return_acquire(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_return_acquire(v); > -} > - > -static __always_inline long > -atomic_long_inc_return_release(atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_return_release(v); > -} > - > -static __always_inline long > -atomic_long_inc_return_relaxed(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_return_relaxed(v); > -} > - > -static __always_inline long > -atomic_long_fetch_inc(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_inc(v); > -} > - > -static __always_inline long > -atomic_long_fetch_inc_acquire(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_inc_acquire(v); > -} > - > -static __always_inline long > -atomic_long_fetch_inc_release(atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_inc_release(v); > -} > - > -static __always_inline long > -atomic_long_fetch_inc_relaxed(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_inc_relaxed(v); > -} > - > -static __always_inline void > -atomic_long_dec(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_dec(v); > -} > - > -static __always_inline long > -atomic_long_dec_return(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_return(v); > -} > - > -static __always_inline long > -atomic_long_dec_return_acquire(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_return_acquire(v); > -} > - > -static __always_inline long > -atomic_long_dec_return_release(atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_return_release(v); > -} > - > -static __always_inline long > -atomic_long_dec_return_relaxed(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_return_relaxed(v); > -} > - > -static __always_inline long > -atomic_long_fetch_dec(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_dec(v); > -} > - > -static __always_inline long > -atomic_long_fetch_dec_acquire(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_dec_acquire(v); > -} > - > -static __always_inline long > -atomic_long_fetch_dec_release(atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_dec_release(v); > -} > - > -static __always_inline long > -atomic_long_fetch_dec_relaxed(atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_dec_relaxed(v); > -} > - > -static __always_inline void > -atomic_long_and(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_and(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_and(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_and(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_and_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_and_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_and_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_and_release(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_and_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_and_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_long_andnot(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_andnot(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_andnot(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_andnot(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_andnot_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_andnot_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_andnot_release(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_andnot_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_long_or(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_or(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_or(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_or(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_or_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_or_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_or_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_or_release(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_or_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_or_relaxed(i, v); > -} > - > -static __always_inline void > -atomic_long_xor(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - arch_atomic_long_xor(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_xor(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_xor(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_xor_acquire(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_xor_acquire(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_xor_release(long i, atomic_long_t *v) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_xor_release(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_xor_relaxed(i, v); > -} > - > -static __always_inline long > -atomic_long_xchg(atomic_long_t *v, long i) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_xchg(v, i); > -} > - > -static __always_inline long > -atomic_long_xchg_acquire(atomic_long_t *v, long i) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_xchg_acquire(v, i); > -} > - > -static __always_inline long > -atomic_long_xchg_release(atomic_long_t *v, long i) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_xchg_release(v, i); > -} > - > -static __always_inline long > -atomic_long_xchg_relaxed(atomic_long_t *v, long i) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_xchg_relaxed(v, i); > -} > - > -static __always_inline long > -atomic_long_cmpxchg(atomic_long_t *v, long old, long new) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_cmpxchg(v, old, new); > -} > - > -static __always_inline long > -atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline long > -atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_cmpxchg_release(v, old, new); > -} > - > -static __always_inline long > -atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_long_try_cmpxchg(v, old, new); > -} > - > -static __always_inline bool > -atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_long_try_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline bool > -atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new) > -{ > - kcsan_release(); > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_long_try_cmpxchg_release(v, old, new); > -} > - > -static __always_inline bool > -atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new) > -{ > - instrument_atomic_read_write(v, sizeof(*v)); > - instrument_atomic_read_write(old, sizeof(*old)); > - return arch_atomic_long_try_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -atomic_long_sub_and_test(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_sub_and_test(i, v); > -} > - > -static __always_inline bool > -atomic_long_dec_and_test(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_and_test(v); > -} > - > -static __always_inline bool > -atomic_long_inc_and_test(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_and_test(v); > -} > - > -static __always_inline bool > -atomic_long_add_negative(long i, atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_add_negative(i, v); > -} > - > -static __always_inline long > -atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_fetch_add_unless(v, a, u); > -} > - > -static __always_inline bool > -atomic_long_add_unless(atomic_long_t *v, long a, long u) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_add_unless(v, a, u); > -} > - > -static __always_inline bool > -atomic_long_inc_not_zero(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_not_zero(v); > -} > - > -static __always_inline bool > -atomic_long_inc_unless_negative(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_inc_unless_negative(v); > -} > - > -static __always_inline bool > -atomic_long_dec_unless_positive(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_unless_positive(v); > -} > - > -static __always_inline long > -atomic_long_dec_if_positive(atomic_long_t *v) > -{ > - kcsan_mb(); > - instrument_atomic_read_write(v, sizeof(*v)); > - return arch_atomic_long_dec_if_positive(v); > -} > - > -#define xchg(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_mb(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_xchg(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define xchg_acquire(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_xchg_acquire(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define xchg_release(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_release(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_xchg_release(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define xchg_relaxed(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_xchg_relaxed(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_mb(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg_acquire(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg_acquire(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg_release(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_release(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg_release(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg_relaxed(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg_relaxed(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg64(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_mb(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg64(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg64_acquire(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg64_acquire(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg64_release(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_release(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg64_release(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg64_relaxed(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define try_cmpxchg(ptr, oldp, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - typeof(oldp) __ai_oldp = (oldp); \ > - kcsan_mb(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > - arch_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > -}) > - > -#define try_cmpxchg_acquire(ptr, oldp, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - typeof(oldp) __ai_oldp = (oldp); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > - arch_try_cmpxchg_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > -}) > - > -#define try_cmpxchg_release(ptr, oldp, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - typeof(oldp) __ai_oldp = (oldp); \ > - kcsan_release(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > - arch_try_cmpxchg_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > -}) > - > -#define try_cmpxchg_relaxed(ptr, oldp, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - typeof(oldp) __ai_oldp = (oldp); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > - arch_try_cmpxchg_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > -}) > - > -#define cmpxchg_local(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg_local(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg64_local(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define sync_cmpxchg(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_mb(); \ > - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > - arch_sync_cmpxchg(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#define cmpxchg_double(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - kcsan_mb(); \ > - instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ > - arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \ > -}) > - > - > -#define cmpxchg_double_local(ptr, ...) \ > -({ \ > - typeof(ptr) __ai_ptr = (ptr); \ > - instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ > - arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \ > -}) > - > -#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ > -// 87c974b93032afd42143613434d1a7788fa598f9 > +/* SPDX-License-Identifier: GPL-2.0-only */ > +#include <generated/atomic-instrumented.h> > diff --git a/include/linux/atomic/atomic-long.h b/include/linux/atomic/atomic-long.h > index 800b8c35992d..2c2986eb66e6 100644 > --- a/include/linux/atomic/atomic-long.h > +++ b/include/linux/atomic/atomic-long.h > @@ -1,1014 +1,2 @@ > -// SPDX-License-Identifier: GPL-2.0 > - > -// Generated by scripts/atomic/gen-atomic-long.sh > -// DO NOT MODIFY THIS FILE DIRECTLY > - > -#ifndef _LINUX_ATOMIC_LONG_H > -#define _LINUX_ATOMIC_LONG_H > - > -#include <linux/compiler.h> > -#include <asm/types.h> > - > -#ifdef CONFIG_64BIT > -typedef atomic64_t atomic_long_t; > -#define ATOMIC_LONG_INIT(i) ATOMIC64_INIT(i) > -#define atomic_long_cond_read_acquire atomic64_cond_read_acquire > -#define atomic_long_cond_read_relaxed atomic64_cond_read_relaxed > -#else > -typedef atomic_t atomic_long_t; > -#define ATOMIC_LONG_INIT(i) ATOMIC_INIT(i) > -#define atomic_long_cond_read_acquire atomic_cond_read_acquire > -#define atomic_long_cond_read_relaxed atomic_cond_read_relaxed > -#endif > - > -#ifdef CONFIG_64BIT > - > -static __always_inline long > -arch_atomic_long_read(const atomic_long_t *v) > -{ > - return arch_atomic64_read(v); > -} > - > -static __always_inline long > -arch_atomic_long_read_acquire(const atomic_long_t *v) > -{ > - return arch_atomic64_read_acquire(v); > -} > - > -static __always_inline void > -arch_atomic_long_set(atomic_long_t *v, long i) > -{ > - arch_atomic64_set(v, i); > -} > - > -static __always_inline void > -arch_atomic_long_set_release(atomic_long_t *v, long i) > -{ > - arch_atomic64_set_release(v, i); > -} > - > -static __always_inline void > -arch_atomic_long_add(long i, atomic_long_t *v) > -{ > - arch_atomic64_add(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return(long i, atomic_long_t *v) > -{ > - return arch_atomic64_add_return(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_add_return_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_add_return_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_add_return_relaxed(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_add(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_add_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_add_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_add_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_sub(long i, atomic_long_t *v) > -{ > - arch_atomic64_sub(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return(long i, atomic_long_t *v) > -{ > - return arch_atomic64_sub_return(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_sub_return_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_sub_return_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_sub_return_relaxed(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_sub(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_sub_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_sub_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_sub_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_inc(atomic_long_t *v) > -{ > - arch_atomic64_inc(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return(atomic_long_t *v) > -{ > - return arch_atomic64_inc_return(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return_acquire(atomic_long_t *v) > -{ > - return arch_atomic64_inc_return_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return_release(atomic_long_t *v) > -{ > - return arch_atomic64_inc_return_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return_relaxed(atomic_long_t *v) > -{ > - return arch_atomic64_inc_return_relaxed(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_inc(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc_acquire(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_inc_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc_release(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_inc_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc_relaxed(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_inc_relaxed(v); > -} > - > -static __always_inline void > -arch_atomic_long_dec(atomic_long_t *v) > -{ > - arch_atomic64_dec(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return(atomic_long_t *v) > -{ > - return arch_atomic64_dec_return(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return_acquire(atomic_long_t *v) > -{ > - return arch_atomic64_dec_return_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return_release(atomic_long_t *v) > -{ > - return arch_atomic64_dec_return_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return_relaxed(atomic_long_t *v) > -{ > - return arch_atomic64_dec_return_relaxed(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_dec(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec_acquire(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_dec_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec_release(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_dec_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec_relaxed(atomic_long_t *v) > -{ > - return arch_atomic64_fetch_dec_relaxed(v); > -} > - > -static __always_inline void > -arch_atomic_long_and(long i, atomic_long_t *v) > -{ > - arch_atomic64_and(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_and(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_and_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_and_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_and_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_andnot(long i, atomic_long_t *v) > -{ > - arch_atomic64_andnot(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_andnot(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_andnot_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_andnot_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_andnot_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_or(long i, atomic_long_t *v) > -{ > - arch_atomic64_or(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_or(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_or_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_or_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_or_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_xor(long i, atomic_long_t *v) > -{ > - arch_atomic64_xor(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_xor(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_xor_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor_release(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_xor_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic64_fetch_xor_relaxed(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_xchg(atomic_long_t *v, long i) > -{ > - return arch_atomic64_xchg(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_xchg_acquire(atomic_long_t *v, long i) > -{ > - return arch_atomic64_xchg_acquire(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_xchg_release(atomic_long_t *v, long i) > -{ > - return arch_atomic64_xchg_release(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_xchg_relaxed(atomic_long_t *v, long i) > -{ > - return arch_atomic64_xchg_relaxed(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic64_cmpxchg(v, old, new); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic64_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic64_cmpxchg_release(v, old, new); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic64_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic64_try_cmpxchg(v, (s64 *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic64_try_cmpxchg_acquire(v, (s64 *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic64_try_cmpxchg_release(v, (s64 *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic64_try_cmpxchg_relaxed(v, (s64 *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_sub_and_test(long i, atomic_long_t *v) > -{ > - return arch_atomic64_sub_and_test(i, v); > -} > - > -static __always_inline bool > -arch_atomic_long_dec_and_test(atomic_long_t *v) > -{ > - return arch_atomic64_dec_and_test(v); > -} > - > -static __always_inline bool > -arch_atomic_long_inc_and_test(atomic_long_t *v) > -{ > - return arch_atomic64_inc_and_test(v); > -} > - > -static __always_inline bool > -arch_atomic_long_add_negative(long i, atomic_long_t *v) > -{ > - return arch_atomic64_add_negative(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u) > -{ > - return arch_atomic64_fetch_add_unless(v, a, u); > -} > - > -static __always_inline bool > -arch_atomic_long_add_unless(atomic_long_t *v, long a, long u) > -{ > - return arch_atomic64_add_unless(v, a, u); > -} > - > -static __always_inline bool > -arch_atomic_long_inc_not_zero(atomic_long_t *v) > -{ > - return arch_atomic64_inc_not_zero(v); > -} > - > -static __always_inline bool > -arch_atomic_long_inc_unless_negative(atomic_long_t *v) > -{ > - return arch_atomic64_inc_unless_negative(v); > -} > - > -static __always_inline bool > -arch_atomic_long_dec_unless_positive(atomic_long_t *v) > -{ > - return arch_atomic64_dec_unless_positive(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_if_positive(atomic_long_t *v) > -{ > - return arch_atomic64_dec_if_positive(v); > -} > - > -#else /* CONFIG_64BIT */ > - > -static __always_inline long > -arch_atomic_long_read(const atomic_long_t *v) > -{ > - return arch_atomic_read(v); > -} > - > -static __always_inline long > -arch_atomic_long_read_acquire(const atomic_long_t *v) > -{ > - return arch_atomic_read_acquire(v); > -} > - > -static __always_inline void > -arch_atomic_long_set(atomic_long_t *v, long i) > -{ > - arch_atomic_set(v, i); > -} > - > -static __always_inline void > -arch_atomic_long_set_release(atomic_long_t *v, long i) > -{ > - arch_atomic_set_release(v, i); > -} > - > -static __always_inline void > -arch_atomic_long_add(long i, atomic_long_t *v) > -{ > - arch_atomic_add(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return(long i, atomic_long_t *v) > -{ > - return arch_atomic_add_return(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_add_return_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_add_return_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_add_return_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_add_return_relaxed(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_add(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_add_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_add_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_add_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_sub(long i, atomic_long_t *v) > -{ > - arch_atomic_sub(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return(long i, atomic_long_t *v) > -{ > - return arch_atomic_sub_return(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_sub_return_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_sub_return_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_sub_return_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_sub_return_relaxed(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_sub(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_sub_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_sub_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_sub_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_inc(atomic_long_t *v) > -{ > - arch_atomic_inc(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return(atomic_long_t *v) > -{ > - return arch_atomic_inc_return(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return_acquire(atomic_long_t *v) > -{ > - return arch_atomic_inc_return_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return_release(atomic_long_t *v) > -{ > - return arch_atomic_inc_return_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_inc_return_relaxed(atomic_long_t *v) > -{ > - return arch_atomic_inc_return_relaxed(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc(atomic_long_t *v) > -{ > - return arch_atomic_fetch_inc(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc_acquire(atomic_long_t *v) > -{ > - return arch_atomic_fetch_inc_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc_release(atomic_long_t *v) > -{ > - return arch_atomic_fetch_inc_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_inc_relaxed(atomic_long_t *v) > -{ > - return arch_atomic_fetch_inc_relaxed(v); > -} > - > -static __always_inline void > -arch_atomic_long_dec(atomic_long_t *v) > -{ > - arch_atomic_dec(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return(atomic_long_t *v) > -{ > - return arch_atomic_dec_return(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return_acquire(atomic_long_t *v) > -{ > - return arch_atomic_dec_return_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return_release(atomic_long_t *v) > -{ > - return arch_atomic_dec_return_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_return_relaxed(atomic_long_t *v) > -{ > - return arch_atomic_dec_return_relaxed(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec(atomic_long_t *v) > -{ > - return arch_atomic_fetch_dec(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec_acquire(atomic_long_t *v) > -{ > - return arch_atomic_fetch_dec_acquire(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec_release(atomic_long_t *v) > -{ > - return arch_atomic_fetch_dec_release(v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_dec_relaxed(atomic_long_t *v) > -{ > - return arch_atomic_fetch_dec_relaxed(v); > -} > - > -static __always_inline void > -arch_atomic_long_and(long i, atomic_long_t *v) > -{ > - arch_atomic_and(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_and(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_and_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_and_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_and_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_and_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_andnot(long i, atomic_long_t *v) > -{ > - arch_atomic_andnot(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_andnot(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_andnot_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_andnot_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_andnot_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_or(long i, atomic_long_t *v) > -{ > - arch_atomic_or(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_or(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_or_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_or_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_or_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_or_relaxed(i, v); > -} > - > -static __always_inline void > -arch_atomic_long_xor(long i, atomic_long_t *v) > -{ > - arch_atomic_xor(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_xor(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor_acquire(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_xor_acquire(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor_release(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_xor_release(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v) > -{ > - return arch_atomic_fetch_xor_relaxed(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_xchg(atomic_long_t *v, long i) > -{ > - return arch_atomic_xchg(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_xchg_acquire(atomic_long_t *v, long i) > -{ > - return arch_atomic_xchg_acquire(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_xchg_release(atomic_long_t *v, long i) > -{ > - return arch_atomic_xchg_release(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_xchg_relaxed(atomic_long_t *v, long i) > -{ > - return arch_atomic_xchg_relaxed(v, i); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic_cmpxchg(v, old, new); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic_cmpxchg_acquire(v, old, new); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic_cmpxchg_release(v, old, new); > -} > - > -static __always_inline long > -arch_atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new) > -{ > - return arch_atomic_cmpxchg_relaxed(v, old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic_try_cmpxchg(v, (int *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic_try_cmpxchg_acquire(v, (int *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic_try_cmpxchg_release(v, (int *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new) > -{ > - return arch_atomic_try_cmpxchg_relaxed(v, (int *)old, new); > -} > - > -static __always_inline bool > -arch_atomic_long_sub_and_test(long i, atomic_long_t *v) > -{ > - return arch_atomic_sub_and_test(i, v); > -} > - > -static __always_inline bool > -arch_atomic_long_dec_and_test(atomic_long_t *v) > -{ > - return arch_atomic_dec_and_test(v); > -} > - > -static __always_inline bool > -arch_atomic_long_inc_and_test(atomic_long_t *v) > -{ > - return arch_atomic_inc_and_test(v); > -} > - > -static __always_inline bool > -arch_atomic_long_add_negative(long i, atomic_long_t *v) > -{ > - return arch_atomic_add_negative(i, v); > -} > - > -static __always_inline long > -arch_atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u) > -{ > - return arch_atomic_fetch_add_unless(v, a, u); > -} > - > -static __always_inline bool > -arch_atomic_long_add_unless(atomic_long_t *v, long a, long u) > -{ > - return arch_atomic_add_unless(v, a, u); > -} > - > -static __always_inline bool > -arch_atomic_long_inc_not_zero(atomic_long_t *v) > -{ > - return arch_atomic_inc_not_zero(v); > -} > - > -static __always_inline bool > -arch_atomic_long_inc_unless_negative(atomic_long_t *v) > -{ > - return arch_atomic_inc_unless_negative(v); > -} > - > -static __always_inline bool > -arch_atomic_long_dec_unless_positive(atomic_long_t *v) > -{ > - return arch_atomic_dec_unless_positive(v); > -} > - > -static __always_inline long > -arch_atomic_long_dec_if_positive(atomic_long_t *v) > -{ > - return arch_atomic_dec_if_positive(v); > -} > - > -#endif /* CONFIG_64BIT */ > -#endif /* _LINUX_ATOMIC_LONG_H */ > -// e8f0e08ff072b74d180eabe2ad001282b38c2c88 > +/* SPDX-License-Identifier: GPL-2.0-only */ > +#include <generated/atomic-long.h> > diff --git a/include/linux/atomic/atomic-arch-fallback.h b/scripts/atomic/atomic-arch-fallback.h_shipped > similarity index 99% > copy from include/linux/atomic/atomic-arch-fallback.h > copy to scripts/atomic/atomic-arch-fallback.h_shipped > index 6db58d180866..357627e3e304 100644 > --- a/include/linux/atomic/atomic-arch-fallback.h > +++ b/scripts/atomic/atomic-arch-fallback.h_shipped > @@ -2386,4 +2386,3 @@ arch_atomic64_dec_if_positive(atomic64_t *v) > #endif > > #endif /* _LINUX_ATOMIC_FALLBACK_H */ > -// 8e2cc06bc0d2c0967d2f8424762bd48555ee40ae > diff --git a/include/linux/atomic/atomic-instrumented.h b/scripts/atomic/atomic-instrumented.h_shipped > similarity index 99% > copy from include/linux/atomic/atomic-instrumented.h > copy to scripts/atomic/atomic-instrumented.h_shipped > index 5d69b143c28e..89cde1f1eb67 100644 > --- a/include/linux/atomic/atomic-instrumented.h > +++ b/scripts/atomic/atomic-instrumented.h_shipped > @@ -2045,4 +2045,3 @@ atomic_long_dec_if_positive(atomic_long_t *v) > }) > > #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ > -// 87c974b93032afd42143613434d1a7788fa598f9 > diff --git a/include/linux/atomic/atomic-long.h b/scripts/atomic/atomic-long.h_shipped > similarity index 99% > copy from include/linux/atomic/atomic-long.h > copy to scripts/atomic/atomic-long.h_shipped > index 800b8c35992d..411d16a7f4eb 100644 > --- a/include/linux/atomic/atomic-long.h > +++ b/scripts/atomic/atomic-long.h_shipped > @@ -1011,4 +1011,3 @@ arch_atomic_long_dec_if_positive(atomic_long_t *v) > > #endif /* CONFIG_64BIT */ > #endif /* _LINUX_ATOMIC_LONG_H */ > -// e8f0e08ff072b74d180eabe2ad001282b38c2c88 > diff --git a/scripts/atomic/check-atomics.sh b/scripts/atomic/check-atomics.sh > deleted file mode 100755 > index 0e7bab3eb0d1..000000000000 > --- a/scripts/atomic/check-atomics.sh > +++ /dev/null > @@ -1,33 +0,0 @@ > -#!/bin/sh > -# SPDX-License-Identifier: GPL-2.0 > -# > -# Check if atomic headers are up-to-date > - > -ATOMICDIR=$(dirname $0) > -ATOMICTBL=${ATOMICDIR}/atomics.tbl > -LINUXDIR=${ATOMICDIR}/../.. > - > -echo '' | sha1sum - > /dev/null 2>&1 > -if [ $? -ne 0 ]; then > - printf "sha1sum not available, skipping atomic header checks.\n" > - exit 0 > -fi > - > -cat <<EOF | > -linux/atomic/atomic-instrumented.h > -linux/atomic/atomic-long.h > -linux/atomic/atomic-arch-fallback.h > -EOF > -while read header; do > - OLDSUM="$(tail -n 1 ${LINUXDIR}/include/${header})" > - OLDSUM="${OLDSUM#// }" > - > - NEWSUM="$(sed '$d' ${LINUXDIR}/include/${header} | sha1sum)" > - NEWSUM="${NEWSUM%% *}" > - > - if [ "${OLDSUM}" != "${NEWSUM}" ]; then > - printf "warning: generated include/${header} has been modified.\n" > - fi > -done > - > -exit 0 > diff --git a/scripts/atomic/gen-atomics.sh b/scripts/atomic/gen-atomics.sh > index 5b98a8307693..ea90b7ac8f2c 100755 > --- a/scripts/atomic/gen-atomics.sh > +++ b/scripts/atomic/gen-atomics.sh > @@ -5,16 +5,12 @@ > > ATOMICDIR=$(dirname $0) > ATOMICTBL=${ATOMICDIR}/atomics.tbl > -LINUXDIR=${ATOMICDIR}/../.. > > cat <<EOF | > -gen-atomic-instrumented.sh linux/atomic/atomic-instrumented.h > -gen-atomic-long.sh linux/atomic/atomic-long.h > -gen-atomic-fallback.sh linux/atomic/atomic-arch-fallback.h > +gen-atomic-instrumented.sh atomic-instrumented.h > +gen-atomic-long.sh atomic-long.h > +gen-atomic-fallback.sh atomic-arch-fallback.h > EOF > -while read script header args; do > - /bin/sh ${ATOMICDIR}/${script} ${ATOMICTBL} ${args} > ${LINUXDIR}/include/${header} > - HASH="$(sha1sum ${LINUXDIR}/include/${header})" > - HASH="${HASH%% *}" > - printf "// %s\n" "${HASH}" >> ${LINUXDIR}/include/${header} > +while read script header; do > + /bin/sh ${ATOMICDIR}/${script} ${ATOMICTBL} > ${ATOMICDIR}/${header}_shipped > done > -- > 2.32.0 >
| |