mirror of
https://mirrors.bfsu.edu.cn/git/linux.git
synced 2024-11-11 12:28:41 +08:00
Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking/atomics update from Thomas Gleixner: "The locking, atomics and memory model brains delivered: - A larger update to the atomics code which reworks the ordering barriers, consolidates the atomic primitives, provides the new atomic64_fetch_add_unless() primitive and cleans up the include hell. - Simplify cmpxchg() instrumentation and add instrumentation for xchg() and cmpxchg_double(). - Updates to the memory model and documentation" * 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (48 commits) locking/atomics: Rework ordering barriers locking/atomics: Instrument cmpxchg_double*() locking/atomics: Instrument xchg() locking/atomics: Simplify cmpxchg() instrumentation locking/atomics/x86: Reduce arch_cmpxchg64*() instrumentation tools/memory-model: Rename litmus tests to comply to norm7 tools/memory-model/Documentation: Fix typo, smb->smp sched/Documentation: Update wake_up() & co. memory-barrier guarantees locking/spinlock, sched/core: Clarify requirements for smp_mb__after_spinlock() sched/core: Use smp_mb() in wake_woken_function() tools/memory-model: Add informal LKMM documentation to MAINTAINERS locking/atomics/Documentation: Describe atomic_set() as a write operation tools/memory-model: Make scripts executable tools/memory-model: Remove ACCESS_ONCE() from model tools/memory-model: Remove ACCESS_ONCE() from recipes locking/memory-barriers.txt/kokr: Update Korean translation to fix broken DMA vs. MMIO ordering example MAINTAINERS: Add Daniel Lustig as an LKMM reviewer tools/memory-model: Fix ISA2+pooncelock+pooncelock+pombonce name tools/memory-model: Add litmus test for full multicopy atomicity locking/refcount: Always allow checked forms ...
This commit is contained in:
commit
de5d1b39ea
@ -29,7 +29,7 @@ updated by one CPU, local_t is probably more appropriate. Please see
|
||||
local_t.
|
||||
|
||||
The first operations to implement for atomic_t's are the initializers and
|
||||
plain reads. ::
|
||||
plain writes. ::
|
||||
|
||||
#define ATOMIC_INIT(i) { (i) }
|
||||
#define atomic_set(v, i) ((v)->counter = (i))
|
||||
|
@ -2179,32 +2179,41 @@ or:
|
||||
event_indicated = 1;
|
||||
wake_up_process(event_daemon);
|
||||
|
||||
A write memory barrier is implied by wake_up() and co. if and only if they
|
||||
wake something up. The barrier occurs before the task state is cleared, and so
|
||||
sits between the STORE to indicate the event and the STORE to set TASK_RUNNING:
|
||||
A general memory barrier is executed by wake_up() if it wakes something up.
|
||||
If it doesn't wake anything up then a memory barrier may or may not be
|
||||
executed; you must not rely on it. The barrier occurs before the task state
|
||||
is accessed, in particular, it sits between the STORE to indicate the event
|
||||
and the STORE to set TASK_RUNNING:
|
||||
|
||||
CPU 1 CPU 2
|
||||
CPU 1 (Sleeper) CPU 2 (Waker)
|
||||
=============================== ===============================
|
||||
set_current_state(); STORE event_indicated
|
||||
smp_store_mb(); wake_up();
|
||||
STORE current->state <write barrier>
|
||||
<general barrier> STORE current->state
|
||||
LOAD event_indicated
|
||||
STORE current->state ...
|
||||
<general barrier> <general barrier>
|
||||
LOAD event_indicated if ((LOAD task->state) & TASK_NORMAL)
|
||||
STORE task->state
|
||||
|
||||
To repeat, this write memory barrier is present if and only if something
|
||||
is actually awakened. To see this, consider the following sequence of
|
||||
events, where X and Y are both initially zero:
|
||||
where "task" is the thread being woken up and it equals CPU 1's "current".
|
||||
|
||||
To repeat, a general memory barrier is guaranteed to be executed by wake_up()
|
||||
if something is actually awakened, but otherwise there is no such guarantee.
|
||||
To see this, consider the following sequence of events, where X and Y are both
|
||||
initially zero:
|
||||
|
||||
CPU 1 CPU 2
|
||||
=============================== ===============================
|
||||
X = 1; STORE event_indicated
|
||||
X = 1; Y = 1;
|
||||
smp_mb(); wake_up();
|
||||
Y = 1; wait_event(wq, Y == 1);
|
||||
wake_up(); load from Y sees 1, no memory barrier
|
||||
load from X might see 0
|
||||
LOAD Y LOAD X
|
||||
|
||||
In contrast, if a wakeup does occur, CPU 2's load from X would be guaranteed
|
||||
to see 1.
|
||||
If a wakeup does occur, one (at least) of the two loads must see 1. If, on
|
||||
the other hand, a wakeup does not occur, both loads might see 0.
|
||||
|
||||
wake_up_process() always executes a general memory barrier. The barrier again
|
||||
occurs before the task state is accessed. In particular, if the wake_up() in
|
||||
the previous snippet were replaced by a call to wake_up_process() then one of
|
||||
the two loads would be guaranteed to see 1.
|
||||
|
||||
The available waker functions include:
|
||||
|
||||
@ -2224,6 +2233,8 @@ The available waker functions include:
|
||||
wake_up_poll();
|
||||
wake_up_process();
|
||||
|
||||
In terms of memory ordering, these functions all provide the same guarantees of
|
||||
a wake_up() (or stronger).
|
||||
|
||||
[!] Note that the memory barriers implied by the sleeper and the waker do _not_
|
||||
order multiple stores before the wake-up with respect to loads of those stored
|
||||
|
@ -1891,22 +1891,22 @@ Mandatory 배리어들은 SMP 시스템에서도 UP 시스템에서도 SMP 효
|
||||
/* 소유권을 수정 */
|
||||
desc->status = DEVICE_OWN;
|
||||
|
||||
/* MMIO 를 통해 디바이스에 공지를 하기 전에 메모리를 동기화 */
|
||||
wmb();
|
||||
|
||||
/* 업데이트된 디스크립터의 디바이스에 공지 */
|
||||
writel(DESC_NOTIFY, doorbell);
|
||||
}
|
||||
|
||||
dma_rmb() 는 디스크립터로부터 데이터를 읽어오기 전에 디바이스가 소유권을
|
||||
내놓았음을 보장하게 하고, dma_wmb() 는 디바이스가 자신이 소유권을 다시
|
||||
가졌음을 보기 전에 디스크립터에 데이터가 쓰였음을 보장합니다. wmb() 는
|
||||
캐시 일관성이 없는 (cache incoherent) MMIO 영역에 쓰기를 시도하기 전에
|
||||
캐시 일관성이 있는 메모리 (cache coherent memory) 쓰기가 완료되었음을
|
||||
보장해주기 위해 필요합니다.
|
||||
내려놓았을 것을 보장하고, dma_wmb() 는 디바이스가 자신이 소유권을 다시
|
||||
가졌음을 보기 전에 디스크립터에 데이터가 쓰였을 것을 보장합니다. 참고로,
|
||||
writel() 을 사용하면 캐시 일관성이 있는 메모리 (cache coherent memory)
|
||||
쓰기가 MMIO 영역에의 쓰기 전에 완료되었을 것을 보장하므로 writel() 앞에
|
||||
wmb() 를 실행할 필요가 없음을 알아두시기 바랍니다. writel() 보다 비용이
|
||||
저렴한 writel_relaxed() 는 이런 보장을 제공하지 않으므로 여기선 사용되지
|
||||
않아야 합니다.
|
||||
|
||||
consistent memory 에 대한 자세한 내용을 위해선 Documentation/DMA-API.txt
|
||||
문서를 참고하세요.
|
||||
writel_relaxed() 와 같은 완화된 I/O 접근자들에 대한 자세한 내용을 위해서는
|
||||
"커널 I/O 배리어의 효과" 섹션을, consistent memory 에 대한 자세한 내용을
|
||||
위해선 Documentation/DMA-API.txt 문서를 참고하세요.
|
||||
|
||||
|
||||
MMIO 쓰기 배리어
|
||||
|
@ -8317,10 +8317,16 @@ M: Jade Alglave <j.alglave@ucl.ac.uk>
|
||||
M: Luc Maranget <luc.maranget@inria.fr>
|
||||
M: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
|
||||
R: Akira Yokosawa <akiyks@gmail.com>
|
||||
R: Daniel Lustig <dlustig@nvidia.com>
|
||||
L: linux-kernel@vger.kernel.org
|
||||
L: linux-arch@vger.kernel.org
|
||||
S: Supported
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
|
||||
F: tools/memory-model/
|
||||
F: Documentation/atomic_bitops.txt
|
||||
F: Documentation/atomic_t.txt
|
||||
F: Documentation/core-api/atomic_ops.rst
|
||||
F: Documentation/core-api/refcount-vs-atomic.rst
|
||||
F: Documentation/memory-barriers.txt
|
||||
|
||||
LINUX SECURITY MODULE (LSM) FRAMEWORK
|
||||
|
@ -18,11 +18,11 @@
|
||||
* To ensure dependency ordering is preserved for the _relaxed and
|
||||
* _release atomics, an smp_read_barrier_depends() is unconditionally
|
||||
* inserted into the _relaxed variants, which are used to build the
|
||||
* barriered versions. To avoid redundant back-to-back fences, we can
|
||||
* define the _acquire and _fence versions explicitly.
|
||||
* barriered versions. Avoid redundant back-to-back fences in the
|
||||
* _acquire and _fence versions.
|
||||
*/
|
||||
#define __atomic_op_acquire(op, args...) op##_relaxed(args)
|
||||
#define __atomic_op_fence __atomic_op_release
|
||||
#define __atomic_acquire_fence()
|
||||
#define __atomic_post_full_fence()
|
||||
|
||||
#define ATOMIC_INIT(i) { (i) }
|
||||
#define ATOMIC64_INIT(i) { (i) }
|
||||
@ -206,7 +206,7 @@ ATOMIC_OPS(xor, xor)
|
||||
#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* atomic_fetch_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
@ -214,7 +214,7 @@ ATOMIC_OPS(xor, xor)
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static __inline__ int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, new, old;
|
||||
smp_mb();
|
||||
@ -235,38 +235,39 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
smp_mb();
|
||||
return old;
|
||||
}
|
||||
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
/**
|
||||
* atomic64_add_unless - add unless the number is a given value
|
||||
* atomic64_fetch_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns true iff @v was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
static __inline__ long atomic64_fetch_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long c, tmp;
|
||||
long c, new, old;
|
||||
smp_mb();
|
||||
__asm__ __volatile__(
|
||||
"1: ldq_l %[tmp],%[mem]\n"
|
||||
" cmpeq %[tmp],%[u],%[c]\n"
|
||||
" addq %[tmp],%[a],%[tmp]\n"
|
||||
"1: ldq_l %[old],%[mem]\n"
|
||||
" cmpeq %[old],%[u],%[c]\n"
|
||||
" addq %[old],%[a],%[new]\n"
|
||||
" bne %[c],2f\n"
|
||||
" stq_c %[tmp],%[mem]\n"
|
||||
" beq %[tmp],3f\n"
|
||||
" stq_c %[new],%[mem]\n"
|
||||
" beq %[new],3f\n"
|
||||
"2:\n"
|
||||
".subsection 2\n"
|
||||
"3: br 1b\n"
|
||||
".previous"
|
||||
: [tmp] "=&r"(tmp), [c] "=&r"(c)
|
||||
: [old] "=&r"(old), [new] "=&r"(new), [c] "=&r"(c)
|
||||
: [mem] "m"(*v), [a] "rI"(a), [u] "rI"(u)
|
||||
: "memory");
|
||||
smp_mb();
|
||||
return !c;
|
||||
return old;
|
||||
}
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
|
||||
/*
|
||||
* atomic64_dec_if_positive - decrement by 1 if old value positive
|
||||
@ -295,31 +296,6 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
|
||||
smp_mb();
|
||||
return old - 1;
|
||||
}
|
||||
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
|
||||
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
|
||||
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
|
||||
|
||||
#define atomic_dec_return(v) atomic_sub_return(1,(v))
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1,(v))
|
||||
|
||||
#define atomic_inc_return(v) atomic_add_return(1,(v))
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1,(v))
|
||||
|
||||
#define atomic_sub_and_test(i,v) (atomic_sub_return((i), (v)) == 0)
|
||||
#define atomic64_sub_and_test(i,v) (atomic64_sub_return((i), (v)) == 0)
|
||||
|
||||
#define atomic_inc_and_test(v) (atomic_add_return(1, (v)) == 0)
|
||||
#define atomic64_inc_and_test(v) (atomic64_add_return(1, (v)) == 0)
|
||||
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
|
||||
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, (v)) == 0)
|
||||
|
||||
#define atomic_inc(v) atomic_add(1,(v))
|
||||
#define atomic64_inc(v) atomic64_add(1,(v))
|
||||
|
||||
#define atomic_dec(v) atomic_sub(1,(v))
|
||||
#define atomic64_dec(v) atomic64_sub(1,(v))
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
|
||||
#endif /* _ALPHA_ATOMIC_H */
|
||||
|
@ -187,7 +187,8 @@ static inline int atomic_fetch_##op(int i, atomic_t *v) \
|
||||
ATOMIC_OPS(add, +=, add)
|
||||
ATOMIC_OPS(sub, -=, sub)
|
||||
|
||||
#define atomic_andnot atomic_andnot
|
||||
#define atomic_andnot atomic_andnot
|
||||
#define atomic_fetch_andnot atomic_fetch_andnot
|
||||
|
||||
#undef ATOMIC_OPS
|
||||
#define ATOMIC_OPS(op, c_op, asm_op) \
|
||||
@ -296,8 +297,6 @@ ATOMIC_OPS(add, +=, CTOP_INST_AADD_DI_R2_R2_R3)
|
||||
ATOMIC_FETCH_OP(op, c_op, asm_op)
|
||||
|
||||
ATOMIC_OPS(and, &=, CTOP_INST_AAND_DI_R2_R2_R3)
|
||||
#define atomic_andnot(mask, v) atomic_and(~(mask), (v))
|
||||
#define atomic_fetch_andnot(mask, v) atomic_fetch_and(~(mask), (v))
|
||||
ATOMIC_OPS(or, |=, CTOP_INST_AOR_DI_R2_R2_R3)
|
||||
ATOMIC_OPS(xor, ^=, CTOP_INST_AXOR_DI_R2_R2_R3)
|
||||
|
||||
@ -308,48 +307,6 @@ ATOMIC_OPS(xor, ^=, CTOP_INST_AXOR_DI_R2_R2_R3)
|
||||
#undef ATOMIC_OP_RETURN
|
||||
#undef ATOMIC_OP
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v
|
||||
*/
|
||||
#define __atomic_add_unless(v, a, u) \
|
||||
({ \
|
||||
int c, old; \
|
||||
\
|
||||
/* \
|
||||
* Explicit full memory barrier needed before/after as \
|
||||
* LLOCK/SCOND thmeselves don't provide any such semantics \
|
||||
*/ \
|
||||
smp_mb(); \
|
||||
\
|
||||
c = atomic_read(v); \
|
||||
while (c != (u) && (old = atomic_cmpxchg((v), c, c + (a))) != c)\
|
||||
c = old; \
|
||||
\
|
||||
smp_mb(); \
|
||||
\
|
||||
c; \
|
||||
})
|
||||
|
||||
#define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)
|
||||
|
||||
#define atomic_inc(v) atomic_add(1, v)
|
||||
#define atomic_dec(v) atomic_sub(1, v)
|
||||
|
||||
#define atomic_inc_and_test(v) (atomic_add_return(1, v) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
|
||||
|
||||
#define atomic_add_negative(i, v) (atomic_add_return(i, v) < 0)
|
||||
|
||||
|
||||
#ifdef CONFIG_GENERIC_ATOMIC64
|
||||
|
||||
#include <asm-generic/atomic64.h>
|
||||
@ -472,7 +429,8 @@ static inline long long atomic64_fetch_##op(long long a, atomic64_t *v) \
|
||||
ATOMIC64_OP_RETURN(op, op1, op2) \
|
||||
ATOMIC64_FETCH_OP(op, op1, op2)
|
||||
|
||||
#define atomic64_andnot atomic64_andnot
|
||||
#define atomic64_andnot atomic64_andnot
|
||||
#define atomic64_fetch_andnot atomic64_fetch_andnot
|
||||
|
||||
ATOMIC64_OPS(add, add.f, adc)
|
||||
ATOMIC64_OPS(sub, sub.f, sbc)
|
||||
@ -559,53 +517,43 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
|
||||
|
||||
return val;
|
||||
}
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
|
||||
/**
|
||||
* atomic64_add_unless - add unless the number is a given value
|
||||
* atomic64_fetch_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* if (v != u) { v += a; ret = 1} else {ret = 0}
|
||||
* Returns 1 iff @v was not @u (i.e. if add actually happened)
|
||||
* Atomically adds @a to @v, if it was not @u.
|
||||
* Returns the old value of @v
|
||||
*/
|
||||
static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
|
||||
static inline long long atomic64_fetch_add_unless(atomic64_t *v, long long a,
|
||||
long long u)
|
||||
{
|
||||
long long val;
|
||||
int op_done;
|
||||
long long old, temp;
|
||||
|
||||
smp_mb();
|
||||
|
||||
__asm__ __volatile__(
|
||||
"1: llockd %0, [%2] \n"
|
||||
" mov %1, 1 \n"
|
||||
" brne %L0, %L4, 2f # continue to add since v != u \n"
|
||||
" breq.d %H0, %H4, 3f # return since v == u \n"
|
||||
" mov %1, 0 \n"
|
||||
"2: \n"
|
||||
" add.f %L0, %L0, %L3 \n"
|
||||
" adc %H0, %H0, %H3 \n"
|
||||
" scondd %0, [%2] \n"
|
||||
" add.f %L1, %L0, %L3 \n"
|
||||
" adc %H1, %H0, %H3 \n"
|
||||
" scondd %1, [%2] \n"
|
||||
" bnz 1b \n"
|
||||
"3: \n"
|
||||
: "=&r"(val), "=&r" (op_done)
|
||||
: "=&r"(old), "=&r" (temp)
|
||||
: "r"(&v->counter), "r"(a), "r"(u)
|
||||
: "cc"); /* memory clobber comes from smp_mb() */
|
||||
|
||||
smp_mb();
|
||||
|
||||
return op_done;
|
||||
return old;
|
||||
}
|
||||
|
||||
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
|
||||
#define atomic64_inc(v) atomic64_add(1LL, (v))
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1LL, (v))
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
|
||||
#define atomic64_dec(v) atomic64_sub(1LL, (v))
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1LL, (v))
|
||||
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1LL, 0LL)
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
|
||||
#endif /* !CONFIG_GENERIC_ATOMIC64 */
|
||||
|
||||
|
@ -130,7 +130,7 @@ static inline int atomic_cmpxchg_relaxed(atomic_t *ptr, int old, int new)
|
||||
}
|
||||
#define atomic_cmpxchg_relaxed atomic_cmpxchg_relaxed
|
||||
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int oldval, newval;
|
||||
unsigned long tmp;
|
||||
@ -156,6 +156,7 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
|
||||
return oldval;
|
||||
}
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
#else /* ARM_ARCH_6 */
|
||||
|
||||
@ -215,15 +216,7 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
|
||||
return ret;
|
||||
}
|
||||
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
|
||||
c = atomic_read(v);
|
||||
while (c != u && (old = atomic_cmpxchg((v), c, c + a)) != c)
|
||||
c = old;
|
||||
return c;
|
||||
}
|
||||
#define atomic_fetch_andnot atomic_fetch_andnot
|
||||
|
||||
#endif /* __LINUX_ARM_ARCH__ */
|
||||
|
||||
@ -254,17 +247,6 @@ ATOMIC_OPS(xor, ^=, eor)
|
||||
|
||||
#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
#define atomic_inc(v) atomic_add(1, v)
|
||||
#define atomic_dec(v) atomic_sub(1, v)
|
||||
|
||||
#define atomic_inc_and_test(v) (atomic_add_return(1, v) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
|
||||
#define atomic_inc_return_relaxed(v) (atomic_add_return_relaxed(1, v))
|
||||
#define atomic_dec_return_relaxed(v) (atomic_sub_return_relaxed(1, v))
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
|
||||
|
||||
#define atomic_add_negative(i,v) (atomic_add_return(i, v) < 0)
|
||||
|
||||
#ifndef CONFIG_GENERIC_ATOMIC64
|
||||
typedef struct {
|
||||
long long counter;
|
||||
@ -494,12 +476,13 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
|
||||
|
||||
return result;
|
||||
}
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
|
||||
static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
|
||||
static inline long long atomic64_fetch_add_unless(atomic64_t *v, long long a,
|
||||
long long u)
|
||||
{
|
||||
long long val;
|
||||
long long oldval, newval;
|
||||
unsigned long tmp;
|
||||
int ret = 1;
|
||||
|
||||
smp_mb();
|
||||
prefetchw(&v->counter);
|
||||
@ -508,33 +491,23 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
|
||||
"1: ldrexd %0, %H0, [%4]\n"
|
||||
" teq %0, %5\n"
|
||||
" teqeq %H0, %H5\n"
|
||||
" moveq %1, #0\n"
|
||||
" beq 2f\n"
|
||||
" adds %Q0, %Q0, %Q6\n"
|
||||
" adc %R0, %R0, %R6\n"
|
||||
" strexd %2, %0, %H0, [%4]\n"
|
||||
" adds %Q1, %Q0, %Q6\n"
|
||||
" adc %R1, %R0, %R6\n"
|
||||
" strexd %2, %1, %H1, [%4]\n"
|
||||
" teq %2, #0\n"
|
||||
" bne 1b\n"
|
||||
"2:"
|
||||
: "=&r" (val), "+r" (ret), "=&r" (tmp), "+Qo" (v->counter)
|
||||
: "=&r" (oldval), "=&r" (newval), "=&r" (tmp), "+Qo" (v->counter)
|
||||
: "r" (&v->counter), "r" (u), "r" (a)
|
||||
: "cc");
|
||||
|
||||
if (ret)
|
||||
if (oldval != u)
|
||||
smp_mb();
|
||||
|
||||
return ret;
|
||||
return oldval;
|
||||
}
|
||||
|
||||
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
|
||||
#define atomic64_inc(v) atomic64_add(1LL, (v))
|
||||
#define atomic64_inc_return_relaxed(v) atomic64_add_return_relaxed(1LL, (v))
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
|
||||
#define atomic64_dec(v) atomic64_sub(1LL, (v))
|
||||
#define atomic64_dec_return_relaxed(v) atomic64_sub_return_relaxed(1LL, (v))
|
||||
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1LL, 0LL)
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
|
||||
#endif /* !CONFIG_GENERIC_ATOMIC64 */
|
||||
#endif
|
||||
|
@ -40,17 +40,6 @@
|
||||
|
||||
#include <asm/cmpxchg.h>
|
||||
|
||||
#define ___atomic_add_unless(v, a, u, sfx) \
|
||||
({ \
|
||||
typeof((v)->counter) c, old; \
|
||||
\
|
||||
c = atomic##sfx##_read(v); \
|
||||
while (c != (u) && \
|
||||
(old = atomic##sfx##_cmpxchg((v), c, c + (a))) != c) \
|
||||
c = old; \
|
||||
c; \
|
||||
})
|
||||
|
||||
#define ATOMIC_INIT(i) { (i) }
|
||||
|
||||
#define atomic_read(v) READ_ONCE((v)->counter)
|
||||
@ -61,21 +50,11 @@
|
||||
#define atomic_add_return_release atomic_add_return_release
|
||||
#define atomic_add_return atomic_add_return
|
||||
|
||||
#define atomic_inc_return_relaxed(v) atomic_add_return_relaxed(1, (v))
|
||||
#define atomic_inc_return_acquire(v) atomic_add_return_acquire(1, (v))
|
||||
#define atomic_inc_return_release(v) atomic_add_return_release(1, (v))
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
|
||||
#define atomic_sub_return_relaxed atomic_sub_return_relaxed
|
||||
#define atomic_sub_return_acquire atomic_sub_return_acquire
|
||||
#define atomic_sub_return_release atomic_sub_return_release
|
||||
#define atomic_sub_return atomic_sub_return
|
||||
|
||||
#define atomic_dec_return_relaxed(v) atomic_sub_return_relaxed(1, (v))
|
||||
#define atomic_dec_return_acquire(v) atomic_sub_return_acquire(1, (v))
|
||||
#define atomic_dec_return_release(v) atomic_sub_return_release(1, (v))
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
|
||||
#define atomic_fetch_add_relaxed atomic_fetch_add_relaxed
|
||||
#define atomic_fetch_add_acquire atomic_fetch_add_acquire
|
||||
#define atomic_fetch_add_release atomic_fetch_add_release
|
||||
@ -119,13 +98,6 @@
|
||||
cmpxchg_release(&((v)->counter), (old), (new))
|
||||
#define atomic_cmpxchg(v, old, new) cmpxchg(&((v)->counter), (old), (new))
|
||||
|
||||
#define atomic_inc(v) atomic_add(1, (v))
|
||||
#define atomic_dec(v) atomic_sub(1, (v))
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return((i), (v)) == 0)
|
||||
#define atomic_add_negative(i, v) (atomic_add_return((i), (v)) < 0)
|
||||
#define __atomic_add_unless(v, a, u) ___atomic_add_unless(v, a, u,)
|
||||
#define atomic_andnot atomic_andnot
|
||||
|
||||
/*
|
||||
@ -140,21 +112,11 @@
|
||||
#define atomic64_add_return_release atomic64_add_return_release
|
||||
#define atomic64_add_return atomic64_add_return
|
||||
|
||||
#define atomic64_inc_return_relaxed(v) atomic64_add_return_relaxed(1, (v))
|
||||
#define atomic64_inc_return_acquire(v) atomic64_add_return_acquire(1, (v))
|
||||
#define atomic64_inc_return_release(v) atomic64_add_return_release(1, (v))
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
|
||||
|
||||
#define atomic64_sub_return_relaxed atomic64_sub_return_relaxed
|
||||
#define atomic64_sub_return_acquire atomic64_sub_return_acquire
|
||||
#define atomic64_sub_return_release atomic64_sub_return_release
|
||||
#define atomic64_sub_return atomic64_sub_return
|
||||
|
||||
#define atomic64_dec_return_relaxed(v) atomic64_sub_return_relaxed(1, (v))
|
||||
#define atomic64_dec_return_acquire(v) atomic64_sub_return_acquire(1, (v))
|
||||
#define atomic64_dec_return_release(v) atomic64_sub_return_release(1, (v))
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
|
||||
|
||||
#define atomic64_fetch_add_relaxed atomic64_fetch_add_relaxed
|
||||
#define atomic64_fetch_add_acquire atomic64_fetch_add_acquire
|
||||
#define atomic64_fetch_add_release atomic64_fetch_add_release
|
||||
@ -195,16 +157,9 @@
|
||||
#define atomic64_cmpxchg_release atomic_cmpxchg_release
|
||||
#define atomic64_cmpxchg atomic_cmpxchg
|
||||
|
||||
#define atomic64_inc(v) atomic64_add(1, (v))
|
||||
#define atomic64_dec(v) atomic64_sub(1, (v))
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
#define atomic64_dec_and_test(v) (atomic64_dec_return(v) == 0)
|
||||
#define atomic64_sub_and_test(i, v) (atomic64_sub_return((i), (v)) == 0)
|
||||
#define atomic64_add_negative(i, v) (atomic64_add_return((i), (v)) < 0)
|
||||
#define atomic64_add_unless(v, a, u) (___atomic_add_unless(v, a, u, 64) != u)
|
||||
#define atomic64_andnot atomic64_andnot
|
||||
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
|
||||
#endif
|
||||
#endif
|
||||
|
@ -17,22 +17,11 @@
|
||||
#define __ASM_BITOPS_H
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <asm/barrier.h>
|
||||
|
||||
#ifndef _LINUX_BITOPS_H
|
||||
#error only <linux/bitops.h> can be included directly
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Little endian assembly atomic bitops.
|
||||
*/
|
||||
extern void set_bit(int nr, volatile unsigned long *p);
|
||||
extern void clear_bit(int nr, volatile unsigned long *p);
|
||||
extern void change_bit(int nr, volatile unsigned long *p);
|
||||
extern int test_and_set_bit(int nr, volatile unsigned long *p);
|
||||
extern int test_and_clear_bit(int nr, volatile unsigned long *p);
|
||||
extern int test_and_change_bit(int nr, volatile unsigned long *p);
|
||||
|
||||
#include <asm-generic/bitops/builtin-__ffs.h>
|
||||
#include <asm-generic/bitops/builtin-ffs.h>
|
||||
#include <asm-generic/bitops/builtin-__fls.h>
|
||||
@ -44,15 +33,11 @@ extern int test_and_change_bit(int nr, volatile unsigned long *p);
|
||||
|
||||
#include <asm-generic/bitops/sched.h>
|
||||
#include <asm-generic/bitops/hweight.h>
|
||||
#include <asm-generic/bitops/lock.h>
|
||||
|
||||
#include <asm-generic/bitops/atomic.h>
|
||||
#include <asm-generic/bitops/lock.h>
|
||||
#include <asm-generic/bitops/non-atomic.h>
|
||||
#include <asm-generic/bitops/le.h>
|
||||
|
||||
/*
|
||||
* Ext2 is defined to use little-endian byte ordering.
|
||||
*/
|
||||
#define ext2_set_bit_atomic(lock, nr, p) test_and_set_bit_le(nr, p)
|
||||
#define ext2_clear_bit_atomic(lock, nr, p) test_and_clear_bit_le(nr, p)
|
||||
#include <asm-generic/bitops/ext2-atomic-setbit.h>
|
||||
|
||||
#endif /* __ASM_BITOPS_H */
|
||||
|
@ -1,5 +1,5 @@
|
||||
# SPDX-License-Identifier: GPL-2.0
|
||||
lib-y := bitops.o clear_user.o delay.o copy_from_user.o \
|
||||
lib-y := clear_user.o delay.o copy_from_user.o \
|
||||
copy_to_user.o copy_in_user.o copy_page.o \
|
||||
clear_page.o memchr.o memcpy.o memmove.o memset.o \
|
||||
memcmp.o strcmp.o strncmp.o strlen.o strnlen.o \
|
||||
|
@ -1,76 +0,0 @@
|
||||
/*
|
||||
* Based on arch/arm/lib/bitops.h
|
||||
*
|
||||
* Copyright (C) 2013 ARM Ltd.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
#include <linux/linkage.h>
|
||||
#include <asm/assembler.h>
|
||||
#include <asm/lse.h>
|
||||
|
||||
/*
|
||||
* x0: bits 5:0 bit offset
|
||||
* bits 31:6 word offset
|
||||
* x1: address
|
||||
*/
|
||||
.macro bitop, name, llsc, lse
|
||||
ENTRY( \name )
|
||||
and w3, w0, #63 // Get bit offset
|
||||
eor w0, w0, w3 // Clear low bits
|
||||
mov x2, #1
|
||||
add x1, x1, x0, lsr #3 // Get word offset
|
||||
alt_lse " prfm pstl1strm, [x1]", "nop"
|
||||
lsl x3, x2, x3 // Create mask
|
||||
|
||||
alt_lse "1: ldxr x2, [x1]", "\lse x3, [x1]"
|
||||
alt_lse " \llsc x2, x2, x3", "nop"
|
||||
alt_lse " stxr w0, x2, [x1]", "nop"
|
||||
alt_lse " cbnz w0, 1b", "nop"
|
||||
|
||||
ret
|
||||
ENDPROC(\name )
|
||||
.endm
|
||||
|
||||
.macro testop, name, llsc, lse
|
||||
ENTRY( \name )
|
||||
and w3, w0, #63 // Get bit offset
|
||||
eor w0, w0, w3 // Clear low bits
|
||||
mov x2, #1
|
||||
add x1, x1, x0, lsr #3 // Get word offset
|
||||
alt_lse " prfm pstl1strm, [x1]", "nop"
|
||||
lsl x4, x2, x3 // Create mask
|
||||
|
||||
alt_lse "1: ldxr x2, [x1]", "\lse x4, x2, [x1]"
|
||||
lsr x0, x2, x3
|
||||
alt_lse " \llsc x2, x2, x4", "nop"
|
||||
alt_lse " stlxr w5, x2, [x1]", "nop"
|
||||
alt_lse " cbnz w5, 1b", "nop"
|
||||
alt_lse " dmb ish", "nop"
|
||||
|
||||
and x0, x0, #1
|
||||
ret
|
||||
ENDPROC(\name )
|
||||
.endm
|
||||
|
||||
/*
|
||||
* Atomic bit operations.
|
||||
*/
|
||||
bitop change_bit, eor, steor
|
||||
bitop clear_bit, bic, stclr
|
||||
bitop set_bit, orr, stset
|
||||
|
||||
testop test_and_change_bit, eor, ldeoral
|
||||
testop test_and_clear_bit, bic, ldclral
|
||||
testop test_and_set_bit, orr, ldsetal
|
@ -2,8 +2,10 @@
|
||||
#ifndef __ARCH_H8300_ATOMIC__
|
||||
#define __ARCH_H8300_ATOMIC__
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/types.h>
|
||||
#include <asm/cmpxchg.h>
|
||||
#include <asm/irqflags.h>
|
||||
|
||||
/*
|
||||
* Atomic operations that C can't guarantee us. Useful for
|
||||
@ -15,8 +17,6 @@
|
||||
#define atomic_read(v) READ_ONCE((v)->counter)
|
||||
#define atomic_set(v, i) WRITE_ONCE(((v)->counter), (i))
|
||||
|
||||
#include <linux/kernel.h>
|
||||
|
||||
#define ATOMIC_OP_RETURN(op, c_op) \
|
||||
static inline int atomic_##op##_return(int i, atomic_t *v) \
|
||||
{ \
|
||||
@ -69,18 +69,6 @@ ATOMIC_OPS(sub, -=)
|
||||
#undef ATOMIC_OP_RETURN
|
||||
#undef ATOMIC_OP
|
||||
|
||||
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
|
||||
|
||||
#define atomic_inc_return(v) atomic_add_return(1, v)
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, v)
|
||||
|
||||
#define atomic_inc(v) (void)atomic_inc_return(v)
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
|
||||
#define atomic_dec(v) (void)atomic_dec_return(v)
|
||||
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
|
||||
|
||||
static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
|
||||
{
|
||||
int ret;
|
||||
@ -94,7 +82,7 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
|
||||
return ret;
|
||||
}
|
||||
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int ret;
|
||||
h8300flags flags;
|
||||
@ -106,5 +94,6 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
arch_local_irq_restore(flags);
|
||||
return ret;
|
||||
}
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
#endif /* __ARCH_H8300_ATOMIC __ */
|
||||
|
@ -164,7 +164,7 @@ ATOMIC_OPS(xor)
|
||||
#undef ATOMIC_OP
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* atomic_fetch_add_unless - add unless the number is a given value
|
||||
* @v: pointer to value
|
||||
* @a: amount to add
|
||||
* @u: unless value is equal to u
|
||||
@ -173,7 +173,7 @@ ATOMIC_OPS(xor)
|
||||
*
|
||||
*/
|
||||
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int __oldval;
|
||||
register int tmp;
|
||||
@ -196,18 +196,6 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
);
|
||||
return __oldval;
|
||||
}
|
||||
|
||||
#define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)
|
||||
|
||||
#define atomic_inc(v) atomic_add(1, (v))
|
||||
#define atomic_dec(v) atomic_sub(1, (v))
|
||||
|
||||
#define atomic_inc_and_test(v) (atomic_add_return(1, (v)) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, (v)) == 0)
|
||||
#define atomic_add_negative(i, v) (atomic_add_return(i, (v)) < 0)
|
||||
|
||||
#define atomic_inc_return(v) (atomic_add_return(1, v))
|
||||
#define atomic_dec_return(v) (atomic_sub_return(1, v))
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
#endif
|
||||
|
@ -215,91 +215,10 @@ ATOMIC64_FETCH_OP(xor, ^)
|
||||
(cmpxchg(&((v)->counter), old, new))
|
||||
#define atomic64_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
|
||||
static __inline__ long atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long c, old;
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c != (u);
|
||||
}
|
||||
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
|
||||
static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
long c, old, dec;
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
dec = c - 1;
|
||||
if (unlikely(dec < 0))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, dec);
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return dec;
|
||||
}
|
||||
|
||||
/*
|
||||
* Atomically add I to V and return TRUE if the resulting value is
|
||||
* negative.
|
||||
*/
|
||||
static __inline__ int
|
||||
atomic_add_negative (int i, atomic_t *v)
|
||||
{
|
||||
return atomic_add_return(i, v) < 0;
|
||||
}
|
||||
|
||||
static __inline__ long
|
||||
atomic64_add_negative (__s64 i, atomic64_t *v)
|
||||
{
|
||||
return atomic64_add_return(i, v) < 0;
|
||||
}
|
||||
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
|
||||
|
||||
#define atomic_sub_and_test(i,v) (atomic_sub_return((i), (v)) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
|
||||
#define atomic_inc_and_test(v) (atomic_add_return(1, (v)) == 0)
|
||||
#define atomic64_sub_and_test(i,v) (atomic64_sub_return((i), (v)) == 0)
|
||||
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, (v)) == 0)
|
||||
#define atomic64_inc_and_test(v) (atomic64_add_return(1, (v)) == 0)
|
||||
|
||||
#define atomic_add(i,v) (void)atomic_add_return((i), (v))
|
||||
#define atomic_sub(i,v) (void)atomic_sub_return((i), (v))
|
||||
#define atomic_inc(v) atomic_add(1, (v))
|
||||
#define atomic_dec(v) atomic_sub(1, (v))
|
||||
|
||||
#define atomic64_add(i,v) (void)atomic64_add_return((i), (v))
|
||||
#define atomic64_sub(i,v) (void)atomic64_sub_return((i), (v))
|
||||
#define atomic64_inc(v) atomic64_add(1, (v))
|
||||
#define atomic64_dec(v) atomic64_sub(1, (v))
|
||||
|
||||
#endif /* _ASM_IA64_ATOMIC_H */
|
||||
|
@ -126,11 +126,13 @@ static inline void atomic_inc(atomic_t *v)
|
||||
{
|
||||
__asm__ __volatile__("addql #1,%0" : "+m" (*v));
|
||||
}
|
||||
#define atomic_inc atomic_inc
|
||||
|
||||
static inline void atomic_dec(atomic_t *v)
|
||||
{
|
||||
__asm__ __volatile__("subql #1,%0" : "+m" (*v));
|
||||
}
|
||||
#define atomic_dec atomic_dec
|
||||
|
||||
static inline int atomic_dec_and_test(atomic_t *v)
|
||||
{
|
||||
@ -138,6 +140,7 @@ static inline int atomic_dec_and_test(atomic_t *v)
|
||||
__asm__ __volatile__("subql #1,%1; seq %0" : "=d" (c), "+m" (*v));
|
||||
return c != 0;
|
||||
}
|
||||
#define atomic_dec_and_test atomic_dec_and_test
|
||||
|
||||
static inline int atomic_dec_and_test_lt(atomic_t *v)
|
||||
{
|
||||
@ -155,6 +158,7 @@ static inline int atomic_inc_and_test(atomic_t *v)
|
||||
__asm__ __volatile__("addql #1,%1; seq %0" : "=d" (c), "+m" (*v));
|
||||
return c != 0;
|
||||
}
|
||||
#define atomic_inc_and_test atomic_inc_and_test
|
||||
|
||||
#ifdef CONFIG_RMW_INSNS
|
||||
|
||||
@ -190,9 +194,6 @@ static inline int atomic_xchg(atomic_t *v, int new)
|
||||
|
||||
#endif /* !CONFIG_RMW_INSNS */
|
||||
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
|
||||
static inline int atomic_sub_and_test(int i, atomic_t *v)
|
||||
{
|
||||
char c;
|
||||
@ -201,6 +202,7 @@ static inline int atomic_sub_and_test(int i, atomic_t *v)
|
||||
: ASM_DI (i));
|
||||
return c != 0;
|
||||
}
|
||||
#define atomic_sub_and_test atomic_sub_and_test
|
||||
|
||||
static inline int atomic_add_negative(int i, atomic_t *v)
|
||||
{
|
||||
@ -210,20 +212,6 @@ static inline int atomic_add_negative(int i, atomic_t *v)
|
||||
: ASM_DI (i));
|
||||
return c != 0;
|
||||
}
|
||||
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
#define atomic_add_negative atomic_add_negative
|
||||
|
||||
#endif /* __ARCH_M68K_ATOMIC __ */
|
||||
|
@ -519,12 +519,16 @@ static inline int __fls(int x)
|
||||
|
||||
#endif
|
||||
|
||||
/* Simple test-and-set bit locks */
|
||||
#define test_and_set_bit_lock test_and_set_bit
|
||||
#define clear_bit_unlock clear_bit
|
||||
#define __clear_bit_unlock clear_bit_unlock
|
||||
|
||||
#include <asm-generic/bitops/ext2-atomic.h>
|
||||
#include <asm-generic/bitops/le.h>
|
||||
#include <asm-generic/bitops/fls64.h>
|
||||
#include <asm-generic/bitops/sched.h>
|
||||
#include <asm-generic/bitops/hweight.h>
|
||||
#include <asm-generic/bitops/lock.h>
|
||||
#endif /* __KERNEL__ */
|
||||
|
||||
#endif /* _M68K_BITOPS_H */
|
||||
|
@ -274,97 +274,12 @@ static __inline__ int atomic_sub_if_positive(int i, atomic_t * v)
|
||||
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic_xchg(v, new) (xchg(&((v)->counter), (new)))
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
|
||||
/*
|
||||
* atomic_sub_and_test - subtract value from variable and test result
|
||||
* @i: integer value to subtract
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically subtracts @i from @v and returns
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return((i), (v)) == 0)
|
||||
|
||||
/*
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
|
||||
/*
|
||||
* atomic_dec_and_test - decrement by 1 and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically decrements @v by 1 and
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
|
||||
|
||||
/*
|
||||
* atomic_dec_if_positive - decrement by 1 if old value positive
|
||||
* @v: pointer of type atomic_t
|
||||
*/
|
||||
#define atomic_dec_if_positive(v) atomic_sub_if_positive(1, v)
|
||||
|
||||
/*
|
||||
* atomic_inc - increment atomic variable
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define atomic_inc(v) atomic_add(1, (v))
|
||||
|
||||
/*
|
||||
* atomic_dec - decrement and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define atomic_dec(v) atomic_sub(1, (v))
|
||||
|
||||
/*
|
||||
* atomic_add_negative - add and test if negative
|
||||
* @v: pointer of type atomic_t
|
||||
* @i: integer value to add
|
||||
*
|
||||
* Atomically adds @i to @v and returns true
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#define atomic_add_negative(i, v) (atomic_add_return(i, (v)) < 0)
|
||||
|
||||
#ifdef CONFIG_64BIT
|
||||
|
||||
#define ATOMIC64_INIT(i) { (i) }
|
||||
@ -620,99 +535,12 @@ static __inline__ long atomic64_sub_if_positive(long i, atomic64_t * v)
|
||||
((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic64_xchg(v, new) (xchg(&((v)->counter), (new)))
|
||||
|
||||
/**
|
||||
* atomic64_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns true iff @v was not @u.
|
||||
*/
|
||||
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long c, old;
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c != (u);
|
||||
}
|
||||
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
|
||||
|
||||
/*
|
||||
* atomic64_sub_and_test - subtract value from variable and test result
|
||||
* @i: integer value to subtract
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically subtracts @i from @v and returns
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic64_sub_and_test(i, v) (atomic64_sub_return((i), (v)) == 0)
|
||||
|
||||
/*
|
||||
* atomic64_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
|
||||
/*
|
||||
* atomic64_dec_and_test - decrement by 1 and test
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically decrements @v by 1 and
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, (v)) == 0)
|
||||
|
||||
/*
|
||||
* atomic64_dec_if_positive - decrement by 1 if old value positive
|
||||
* @v: pointer of type atomic64_t
|
||||
*/
|
||||
#define atomic64_dec_if_positive(v) atomic64_sub_if_positive(1, v)
|
||||
|
||||
/*
|
||||
* atomic64_inc - increment atomic variable
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define atomic64_inc(v) atomic64_add(1, (v))
|
||||
|
||||
/*
|
||||
* atomic64_dec - decrement and test
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define atomic64_dec(v) atomic64_sub(1, (v))
|
||||
|
||||
/*
|
||||
* atomic64_add_negative - add and test if negative
|
||||
* @v: pointer of type atomic64_t
|
||||
* @i: integer value to add
|
||||
*
|
||||
* Atomically adds @i to @v and returns true
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#define atomic64_add_negative(i, v) (atomic64_add_return(i, (v)) < 0)
|
||||
|
||||
#endif /* CONFIG_64BIT */
|
||||
|
||||
#endif /* _ASM_ATOMIC_H */
|
||||
|
@ -100,7 +100,7 @@ ATOMIC_OP(xor)
|
||||
*
|
||||
* This is often used through atomic_inc_not_zero()
|
||||
*/
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int old, tmp;
|
||||
|
||||
@ -119,7 +119,7 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
|
||||
return old;
|
||||
}
|
||||
#define __atomic_add_unless __atomic_add_unless
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
#include <asm-generic/atomic.h>
|
||||
|
||||
|
@ -16,8 +16,9 @@
|
||||
#ifndef __ASM_OPENRISC_CMPXCHG_H
|
||||
#define __ASM_OPENRISC_CMPXCHG_H
|
||||
|
||||
#include <linux/bits.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/types.h>
|
||||
#include <linux/bitops.h>
|
||||
|
||||
#define __HAVE_ARCH_CMPXCHG 1
|
||||
|
||||
|
@ -77,30 +77,6 @@ static __inline__ int atomic_read(const atomic_t *v)
|
||||
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
#define ATOMIC_OP(op, c_op) \
|
||||
static __inline__ void atomic_##op(int i, atomic_t *v) \
|
||||
{ \
|
||||
@ -160,28 +136,6 @@ ATOMIC_OPS(xor, ^=)
|
||||
#undef ATOMIC_OP_RETURN
|
||||
#undef ATOMIC_OP
|
||||
|
||||
#define atomic_inc(v) (atomic_add( 1,(v)))
|
||||
#define atomic_dec(v) (atomic_add( -1,(v)))
|
||||
|
||||
#define atomic_inc_return(v) (atomic_add_return( 1,(v)))
|
||||
#define atomic_dec_return(v) (atomic_add_return( -1,(v)))
|
||||
|
||||
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
|
||||
|
||||
/*
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
|
||||
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
|
||||
|
||||
#define atomic_sub_and_test(i,v) (atomic_sub_return((i),(v)) == 0)
|
||||
|
||||
#define ATOMIC_INIT(i) { (i) }
|
||||
|
||||
#ifdef CONFIG_64BIT
|
||||
@ -264,72 +218,11 @@ atomic64_read(const atomic64_t *v)
|
||||
return READ_ONCE((v)->counter);
|
||||
}
|
||||
|
||||
#define atomic64_inc(v) (atomic64_add( 1,(v)))
|
||||
#define atomic64_dec(v) (atomic64_add( -1,(v)))
|
||||
|
||||
#define atomic64_inc_return(v) (atomic64_add_return( 1,(v)))
|
||||
#define atomic64_dec_return(v) (atomic64_add_return( -1,(v)))
|
||||
|
||||
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
|
||||
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
#define atomic64_dec_and_test(v) (atomic64_dec_return(v) == 0)
|
||||
#define atomic64_sub_and_test(i,v) (atomic64_sub_return((i),(v)) == 0)
|
||||
|
||||
/* exported interface */
|
||||
#define atomic64_cmpxchg(v, o, n) \
|
||||
((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic64_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
/**
|
||||
* atomic64_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long c, old;
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c != (u);
|
||||
}
|
||||
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
|
||||
/*
|
||||
* atomic64_dec_if_positive - decrement by 1 if old value positive
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* The function returns the old value of *v minus 1, even if
|
||||
* the atomic variable, v, was not decremented.
|
||||
*/
|
||||
static inline long atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
long c, old, dec;
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
dec = c - 1;
|
||||
if (unlikely(dec < 0))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, dec);
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return dec;
|
||||
}
|
||||
|
||||
#endif /* !CONFIG_64BIT */
|
||||
|
||||
|
||||
|
@ -18,18 +18,11 @@
|
||||
* a "bne-" instruction at the end, so an isync is enough as a acquire barrier
|
||||
* on the platform without lwsync.
|
||||
*/
|
||||
#define __atomic_op_acquire(op, args...) \
|
||||
({ \
|
||||
typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \
|
||||
__asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory"); \
|
||||
__ret; \
|
||||
})
|
||||
#define __atomic_acquire_fence() \
|
||||
__asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory")
|
||||
|
||||
#define __atomic_op_release(op, args...) \
|
||||
({ \
|
||||
__asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory"); \
|
||||
op##_relaxed(args); \
|
||||
})
|
||||
#define __atomic_release_fence() \
|
||||
__asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory")
|
||||
|
||||
static __inline__ int atomic_read(const atomic_t *v)
|
||||
{
|
||||
@ -129,8 +122,6 @@ ATOMIC_OPS(xor, xor)
|
||||
#undef ATOMIC_OP_RETURN_RELAXED
|
||||
#undef ATOMIC_OP
|
||||
|
||||
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
|
||||
|
||||
static __inline__ void atomic_inc(atomic_t *v)
|
||||
{
|
||||
int t;
|
||||
@ -145,6 +136,7 @@ static __inline__ void atomic_inc(atomic_t *v)
|
||||
: "r" (&v->counter)
|
||||
: "cc", "xer");
|
||||
}
|
||||
#define atomic_inc atomic_inc
|
||||
|
||||
static __inline__ int atomic_inc_return_relaxed(atomic_t *v)
|
||||
{
|
||||
@ -163,16 +155,6 @@ static __inline__ int atomic_inc_return_relaxed(atomic_t *v)
|
||||
return t;
|
||||
}
|
||||
|
||||
/*
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
|
||||
static __inline__ void atomic_dec(atomic_t *v)
|
||||
{
|
||||
int t;
|
||||
@ -187,6 +169,7 @@ static __inline__ void atomic_dec(atomic_t *v)
|
||||
: "r" (&v->counter)
|
||||
: "cc", "xer");
|
||||
}
|
||||
#define atomic_dec atomic_dec
|
||||
|
||||
static __inline__ int atomic_dec_return_relaxed(atomic_t *v)
|
||||
{
|
||||
@ -218,7 +201,7 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v)
|
||||
#define atomic_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new))
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* atomic_fetch_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
@ -226,13 +209,13 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v)
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static __inline__ int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int t;
|
||||
|
||||
__asm__ __volatile__ (
|
||||
PPC_ATOMIC_ENTRY_BARRIER
|
||||
"1: lwarx %0,0,%1 # __atomic_add_unless\n\
|
||||
"1: lwarx %0,0,%1 # atomic_fetch_add_unless\n\
|
||||
cmpw 0,%0,%3 \n\
|
||||
beq 2f \n\
|
||||
add %0,%2,%0 \n"
|
||||
@ -248,6 +231,7 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
|
||||
return t;
|
||||
}
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
/**
|
||||
* atomic_inc_not_zero - increment unless the number is zero
|
||||
@ -280,9 +264,6 @@ static __inline__ int atomic_inc_not_zero(atomic_t *v)
|
||||
}
|
||||
#define atomic_inc_not_zero(v) atomic_inc_not_zero((v))
|
||||
|
||||
#define atomic_sub_and_test(a, v) (atomic_sub_return((a), (v)) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_dec_return((v)) == 0)
|
||||
|
||||
/*
|
||||
* Atomically test *v and decrement if it is greater than 0.
|
||||
* The function returns the old value of *v minus 1, even if
|
||||
@ -412,8 +393,6 @@ ATOMIC64_OPS(xor, xor)
|
||||
#undef ATOMIC64_OP_RETURN_RELAXED
|
||||
#undef ATOMIC64_OP
|
||||
|
||||
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
|
||||
|
||||
static __inline__ void atomic64_inc(atomic64_t *v)
|
||||
{
|
||||
long t;
|
||||
@ -427,6 +406,7 @@ static __inline__ void atomic64_inc(atomic64_t *v)
|
||||
: "r" (&v->counter)
|
||||
: "cc", "xer");
|
||||
}
|
||||
#define atomic64_inc atomic64_inc
|
||||
|
||||
static __inline__ long atomic64_inc_return_relaxed(atomic64_t *v)
|
||||
{
|
||||
@ -444,16 +424,6 @@ static __inline__ long atomic64_inc_return_relaxed(atomic64_t *v)
|
||||
return t;
|
||||
}
|
||||
|
||||
/*
|
||||
* atomic64_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
|
||||
static __inline__ void atomic64_dec(atomic64_t *v)
|
||||
{
|
||||
long t;
|
||||
@ -467,6 +437,7 @@ static __inline__ void atomic64_dec(atomic64_t *v)
|
||||
: "r" (&v->counter)
|
||||
: "cc", "xer");
|
||||
}
|
||||
#define atomic64_dec atomic64_dec
|
||||
|
||||
static __inline__ long atomic64_dec_return_relaxed(atomic64_t *v)
|
||||
{
|
||||
@ -487,9 +458,6 @@ static __inline__ long atomic64_dec_return_relaxed(atomic64_t *v)
|
||||
#define atomic64_inc_return_relaxed atomic64_inc_return_relaxed
|
||||
#define atomic64_dec_return_relaxed atomic64_dec_return_relaxed
|
||||
|
||||
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
|
||||
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
|
||||
|
||||
/*
|
||||
* Atomically test *v and decrement if it is greater than 0.
|
||||
* The function returns the old value of *v minus 1.
|
||||
@ -513,6 +481,7 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
|
||||
|
||||
return t;
|
||||
}
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
|
||||
#define atomic64_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic64_cmpxchg_relaxed(v, o, n) \
|
||||
@ -524,7 +493,7 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
|
||||
#define atomic64_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new))
|
||||
|
||||
/**
|
||||
* atomic64_add_unless - add unless the number is a given value
|
||||
* atomic64_fetch_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
@ -532,13 +501,13 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
static __inline__ long atomic64_fetch_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long t;
|
||||
|
||||
__asm__ __volatile__ (
|
||||
PPC_ATOMIC_ENTRY_BARRIER
|
||||
"1: ldarx %0,0,%1 # __atomic_add_unless\n\
|
||||
"1: ldarx %0,0,%1 # atomic64_fetch_add_unless\n\
|
||||
cmpd 0,%0,%3 \n\
|
||||
beq 2f \n\
|
||||
add %0,%2,%0 \n"
|
||||
@ -551,8 +520,9 @@ static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
: "r" (&v->counter), "r" (a), "r" (u)
|
||||
: "cc", "memory");
|
||||
|
||||
return t != u;
|
||||
return t;
|
||||
}
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
|
||||
/**
|
||||
* atomic_inc64_not_zero - increment unless the number is zero
|
||||
@ -582,6 +552,7 @@ static __inline__ int atomic64_inc_not_zero(atomic64_t *v)
|
||||
|
||||
return t1 != 0;
|
||||
}
|
||||
#define atomic64_inc_not_zero(v) atomic64_inc_not_zero((v))
|
||||
|
||||
#endif /* __powerpc64__ */
|
||||
|
||||
|
@ -25,18 +25,11 @@
|
||||
|
||||
#define ATOMIC_INIT(i) { (i) }
|
||||
|
||||
#define __atomic_op_acquire(op, args...) \
|
||||
({ \
|
||||
typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \
|
||||
__asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory"); \
|
||||
__ret; \
|
||||
})
|
||||
#define __atomic_acquire_fence() \
|
||||
__asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory")
|
||||
|
||||
#define __atomic_op_release(op, args...) \
|
||||
({ \
|
||||
__asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); \
|
||||
op##_relaxed(args); \
|
||||
})
|
||||
#define __atomic_release_fence() \
|
||||
__asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory");
|
||||
|
||||
static __always_inline int atomic_read(const atomic_t *v)
|
||||
{
|
||||
@ -209,130 +202,8 @@ ATOMIC_OPS(xor, xor, i)
|
||||
#undef ATOMIC_FETCH_OP
|
||||
#undef ATOMIC_OP_RETURN
|
||||
|
||||
/*
|
||||
* The extra atomic operations that are constructed from one of the core
|
||||
* AMO-based operations above (aside from sub, which is easier to fit above).
|
||||
* These are required to perform a full barrier, but they're OK this way
|
||||
* because atomic_*_return is also required to perform a full barrier.
|
||||
*
|
||||
*/
|
||||
#define ATOMIC_OP(op, func_op, comp_op, I, c_type, prefix) \
|
||||
static __always_inline \
|
||||
bool atomic##prefix##_##op(c_type i, atomic##prefix##_t *v) \
|
||||
{ \
|
||||
return atomic##prefix##_##func_op##_return(i, v) comp_op I; \
|
||||
}
|
||||
|
||||
#ifdef CONFIG_GENERIC_ATOMIC64
|
||||
#define ATOMIC_OPS(op, func_op, comp_op, I) \
|
||||
ATOMIC_OP(op, func_op, comp_op, I, int, )
|
||||
#else
|
||||
#define ATOMIC_OPS(op, func_op, comp_op, I) \
|
||||
ATOMIC_OP(op, func_op, comp_op, I, int, ) \
|
||||
ATOMIC_OP(op, func_op, comp_op, I, long, 64)
|
||||
#endif
|
||||
|
||||
ATOMIC_OPS(add_and_test, add, ==, 0)
|
||||
ATOMIC_OPS(sub_and_test, sub, ==, 0)
|
||||
ATOMIC_OPS(add_negative, add, <, 0)
|
||||
|
||||
#undef ATOMIC_OP
|
||||
#undef ATOMIC_OPS
|
||||
|
||||
#define ATOMIC_OP(op, func_op, I, c_type, prefix) \
|
||||
static __always_inline \
|
||||
void atomic##prefix##_##op(atomic##prefix##_t *v) \
|
||||
{ \
|
||||
atomic##prefix##_##func_op(I, v); \
|
||||
}
|
||||
|
||||
#define ATOMIC_FETCH_OP(op, func_op, I, c_type, prefix) \
|
||||
static __always_inline \
|
||||
c_type atomic##prefix##_fetch_##op##_relaxed(atomic##prefix##_t *v) \
|
||||
{ \
|
||||
return atomic##prefix##_fetch_##func_op##_relaxed(I, v); \
|
||||
} \
|
||||
static __always_inline \
|
||||
c_type atomic##prefix##_fetch_##op(atomic##prefix##_t *v) \
|
||||
{ \
|
||||
return atomic##prefix##_fetch_##func_op(I, v); \
|
||||
}
|
||||
|
||||
#define ATOMIC_OP_RETURN(op, asm_op, c_op, I, c_type, prefix) \
|
||||
static __always_inline \
|
||||
c_type atomic##prefix##_##op##_return_relaxed(atomic##prefix##_t *v) \
|
||||
{ \
|
||||
return atomic##prefix##_fetch_##op##_relaxed(v) c_op I; \
|
||||
} \
|
||||
static __always_inline \
|
||||
c_type atomic##prefix##_##op##_return(atomic##prefix##_t *v) \
|
||||
{ \
|
||||
return atomic##prefix##_fetch_##op(v) c_op I; \
|
||||
}
|
||||
|
||||
#ifdef CONFIG_GENERIC_ATOMIC64
|
||||
#define ATOMIC_OPS(op, asm_op, c_op, I) \
|
||||
ATOMIC_OP( op, asm_op, I, int, ) \
|
||||
ATOMIC_FETCH_OP( op, asm_op, I, int, ) \
|
||||
ATOMIC_OP_RETURN(op, asm_op, c_op, I, int, )
|
||||
#else
|
||||
#define ATOMIC_OPS(op, asm_op, c_op, I) \
|
||||
ATOMIC_OP( op, asm_op, I, int, ) \
|
||||
ATOMIC_FETCH_OP( op, asm_op, I, int, ) \
|
||||
ATOMIC_OP_RETURN(op, asm_op, c_op, I, int, ) \
|
||||
ATOMIC_OP( op, asm_op, I, long, 64) \
|
||||
ATOMIC_FETCH_OP( op, asm_op, I, long, 64) \
|
||||
ATOMIC_OP_RETURN(op, asm_op, c_op, I, long, 64)
|
||||
#endif
|
||||
|
||||
ATOMIC_OPS(inc, add, +, 1)
|
||||
ATOMIC_OPS(dec, add, +, -1)
|
||||
|
||||
#define atomic_inc_return_relaxed atomic_inc_return_relaxed
|
||||
#define atomic_dec_return_relaxed atomic_dec_return_relaxed
|
||||
#define atomic_inc_return atomic_inc_return
|
||||
#define atomic_dec_return atomic_dec_return
|
||||
|
||||
#define atomic_fetch_inc_relaxed atomic_fetch_inc_relaxed
|
||||
#define atomic_fetch_dec_relaxed atomic_fetch_dec_relaxed
|
||||
#define atomic_fetch_inc atomic_fetch_inc
|
||||
#define atomic_fetch_dec atomic_fetch_dec
|
||||
|
||||
#ifndef CONFIG_GENERIC_ATOMIC64
|
||||
#define atomic64_inc_return_relaxed atomic64_inc_return_relaxed
|
||||
#define atomic64_dec_return_relaxed atomic64_dec_return_relaxed
|
||||
#define atomic64_inc_return atomic64_inc_return
|
||||
#define atomic64_dec_return atomic64_dec_return
|
||||
|
||||
#define atomic64_fetch_inc_relaxed atomic64_fetch_inc_relaxed
|
||||
#define atomic64_fetch_dec_relaxed atomic64_fetch_dec_relaxed
|
||||
#define atomic64_fetch_inc atomic64_fetch_inc
|
||||
#define atomic64_fetch_dec atomic64_fetch_dec
|
||||
#endif
|
||||
|
||||
#undef ATOMIC_OPS
|
||||
#undef ATOMIC_OP
|
||||
#undef ATOMIC_FETCH_OP
|
||||
#undef ATOMIC_OP_RETURN
|
||||
|
||||
#define ATOMIC_OP(op, func_op, comp_op, I, prefix) \
|
||||
static __always_inline \
|
||||
bool atomic##prefix##_##op(atomic##prefix##_t *v) \
|
||||
{ \
|
||||
return atomic##prefix##_##func_op##_return(v) comp_op I; \
|
||||
}
|
||||
|
||||
ATOMIC_OP(inc_and_test, inc, ==, 0, )
|
||||
ATOMIC_OP(dec_and_test, dec, ==, 0, )
|
||||
#ifndef CONFIG_GENERIC_ATOMIC64
|
||||
ATOMIC_OP(inc_and_test, inc, ==, 0, 64)
|
||||
ATOMIC_OP(dec_and_test, dec, ==, 0, 64)
|
||||
#endif
|
||||
|
||||
#undef ATOMIC_OP
|
||||
|
||||
/* This is required to provide a full barrier on success. */
|
||||
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static __always_inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int prev, rc;
|
||||
|
||||
@ -349,9 +220,10 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
: "memory");
|
||||
return prev;
|
||||
}
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
#ifndef CONFIG_GENERIC_ATOMIC64
|
||||
static __always_inline long __atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
static __always_inline long atomic64_fetch_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long prev, rc;
|
||||
|
||||
@ -368,27 +240,7 @@ static __always_inline long __atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
: "memory");
|
||||
return prev;
|
||||
}
|
||||
|
||||
static __always_inline int atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
return __atomic64_add_unless(v, a, u) != u;
|
||||
}
|
||||
#endif
|
||||
|
||||
/*
|
||||
* The extra atomic operations that are constructed from one of the core
|
||||
* LR/SC-based operations above.
|
||||
*/
|
||||
static __always_inline int atomic_inc_not_zero(atomic_t *v)
|
||||
{
|
||||
return __atomic_add_unless(v, 1, 0);
|
||||
}
|
||||
|
||||
#ifndef CONFIG_GENERIC_ATOMIC64
|
||||
static __always_inline long atomic64_inc_not_zero(atomic64_t *v)
|
||||
{
|
||||
return atomic64_add_unless(v, 1, 0);
|
||||
}
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
#endif
|
||||
|
||||
/*
|
||||
|
@ -55,17 +55,9 @@ static inline void atomic_add(int i, atomic_t *v)
|
||||
__atomic_add(i, &v->counter);
|
||||
}
|
||||
|
||||
#define atomic_add_negative(_i, _v) (atomic_add_return(_i, _v) < 0)
|
||||
#define atomic_inc(_v) atomic_add(1, _v)
|
||||
#define atomic_inc_return(_v) atomic_add_return(1, _v)
|
||||
#define atomic_inc_and_test(_v) (atomic_add_return(1, _v) == 0)
|
||||
#define atomic_sub(_i, _v) atomic_add(-(int)(_i), _v)
|
||||
#define atomic_sub_return(_i, _v) atomic_add_return(-(int)(_i), _v)
|
||||
#define atomic_fetch_sub(_i, _v) atomic_fetch_add(-(int)(_i), _v)
|
||||
#define atomic_sub_and_test(_i, _v) (atomic_sub_return(_i, _v) == 0)
|
||||
#define atomic_dec(_v) atomic_sub(1, _v)
|
||||
#define atomic_dec_return(_v) atomic_sub_return(1, _v)
|
||||
#define atomic_dec_and_test(_v) (atomic_sub_return(1, _v) == 0)
|
||||
|
||||
#define ATOMIC_OPS(op) \
|
||||
static inline void atomic_##op(int i, atomic_t *v) \
|
||||
@ -90,21 +82,6 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
|
||||
return __atomic_cmpxchg(&v->counter, old, new);
|
||||
}
|
||||
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == u))
|
||||
break;
|
||||
old = atomic_cmpxchg(v, c, c + a);
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
#define ATOMIC64_INIT(i) { (i) }
|
||||
|
||||
static inline long atomic64_read(const atomic64_t *v)
|
||||
@ -168,50 +145,8 @@ ATOMIC64_OPS(xor)
|
||||
|
||||
#undef ATOMIC64_OPS
|
||||
|
||||
static inline int atomic64_add_unless(atomic64_t *v, long i, long u)
|
||||
{
|
||||
long c, old;
|
||||
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == u))
|
||||
break;
|
||||
old = atomic64_cmpxchg(v, c, c + i);
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c != u;
|
||||
}
|
||||
|
||||
static inline long atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
long c, old, dec;
|
||||
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
dec = c - 1;
|
||||
if (unlikely(dec < 0))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, dec);
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return dec;
|
||||
}
|
||||
|
||||
#define atomic64_add_negative(_i, _v) (atomic64_add_return(_i, _v) < 0)
|
||||
#define atomic64_inc(_v) atomic64_add(1, _v)
|
||||
#define atomic64_inc_return(_v) atomic64_add_return(1, _v)
|
||||
#define atomic64_inc_and_test(_v) (atomic64_add_return(1, _v) == 0)
|
||||
#define atomic64_sub_return(_i, _v) atomic64_add_return(-(long)(_i), _v)
|
||||
#define atomic64_fetch_sub(_i, _v) atomic64_fetch_add(-(long)(_i), _v)
|
||||
#define atomic64_sub(_i, _v) atomic64_add(-(long)(_i), _v)
|
||||
#define atomic64_sub_and_test(_i, _v) (atomic64_sub_return(_i, _v) == 0)
|
||||
#define atomic64_dec(_v) atomic64_sub(1, _v)
|
||||
#define atomic64_dec_return(_v) atomic64_sub_return(1, _v)
|
||||
#define atomic64_dec_and_test(_v) (atomic64_sub_return(1, _v) == 0)
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
|
||||
#endif /* __ARCH_S390_ATOMIC__ */
|
||||
|
@ -32,44 +32,9 @@
|
||||
#include <asm/atomic-irq.h>
|
||||
#endif
|
||||
|
||||
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
#define atomic_sub_and_test(i,v) (atomic_sub_return((i), (v)) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
|
||||
|
||||
#define atomic_inc(v) atomic_add(1, (v))
|
||||
#define atomic_dec(v) atomic_sub(1, (v))
|
||||
|
||||
#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
|
||||
return c;
|
||||
}
|
||||
|
||||
#endif /* CONFIG_CPU_J2 */
|
||||
|
||||
#endif /* __ASM_SH_ATOMIC_H */
|
||||
|
@ -8,7 +8,8 @@
|
||||
* This work is licensed under the terms of the GNU GPL, version 2. See the
|
||||
* file "COPYING" in the main directory of this archive for more details.
|
||||
*/
|
||||
#include <linux/bitops.h>
|
||||
#include <linux/bits.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <asm/byteorder.h>
|
||||
|
||||
/*
|
||||
|
@ -27,17 +27,17 @@ int atomic_fetch_or(int, atomic_t *);
|
||||
int atomic_fetch_xor(int, atomic_t *);
|
||||
int atomic_cmpxchg(atomic_t *, int, int);
|
||||
int atomic_xchg(atomic_t *, int);
|
||||
int __atomic_add_unless(atomic_t *, int, int);
|
||||
int atomic_fetch_add_unless(atomic_t *, int, int);
|
||||
void atomic_set(atomic_t *, int);
|
||||
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
|
||||
#define atomic_set_release(v, i) atomic_set((v), (i))
|
||||
|
||||
#define atomic_read(v) READ_ONCE((v)->counter)
|
||||
|
||||
#define atomic_add(i, v) ((void)atomic_add_return( (int)(i), (v)))
|
||||
#define atomic_sub(i, v) ((void)atomic_add_return(-(int)(i), (v)))
|
||||
#define atomic_inc(v) ((void)atomic_add_return( 1, (v)))
|
||||
#define atomic_dec(v) ((void)atomic_add_return( -1, (v)))
|
||||
|
||||
#define atomic_and(i, v) ((void)atomic_fetch_and((i), (v)))
|
||||
#define atomic_or(i, v) ((void)atomic_fetch_or((i), (v)))
|
||||
@ -46,22 +46,4 @@ void atomic_set(atomic_t *, int);
|
||||
#define atomic_sub_return(i, v) (atomic_add_return(-(int)(i), (v)))
|
||||
#define atomic_fetch_sub(i, v) (atomic_fetch_add (-(int)(i), (v)))
|
||||
|
||||
#define atomic_inc_return(v) (atomic_add_return( 1, (v)))
|
||||
#define atomic_dec_return(v) (atomic_add_return( -1, (v)))
|
||||
|
||||
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
|
||||
|
||||
/*
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
|
||||
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
|
||||
|
||||
#endif /* !(__ARCH_SPARC_ATOMIC__) */
|
||||
|
@ -50,38 +50,6 @@ ATOMIC_OPS(xor)
|
||||
#undef ATOMIC_OP_RETURN
|
||||
#undef ATOMIC_OP
|
||||
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, v)
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1, v)
|
||||
|
||||
#define atomic_inc_return(v) atomic_add_return(1, v)
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1, v)
|
||||
|
||||
/*
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
|
||||
#define atomic64_sub_and_test(i, v) (atomic64_sub_return(i, v) == 0)
|
||||
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
|
||||
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, v) == 0)
|
||||
|
||||
#define atomic_inc(v) atomic_add(1, v)
|
||||
#define atomic64_inc(v) atomic64_add(1, v)
|
||||
|
||||
#define atomic_dec(v) atomic_sub(1, v)
|
||||
#define atomic64_dec(v) atomic64_sub(1, v)
|
||||
|
||||
#define atomic_add_negative(i, v) (atomic_add_return(i, v) < 0)
|
||||
#define atomic64_add_negative(i, v) (atomic64_add_return(i, v) < 0)
|
||||
|
||||
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
|
||||
|
||||
static inline int atomic_xchg(atomic_t *v, int new)
|
||||
@ -89,42 +57,11 @@ static inline int atomic_xchg(atomic_t *v, int new)
|
||||
return xchg(&v->counter, new);
|
||||
}
|
||||
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
#define atomic64_cmpxchg(v, o, n) \
|
||||
((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic64_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
static inline long atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
long c, old;
|
||||
c = atomic64_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic64_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c != (u);
|
||||
}
|
||||
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
|
||||
long atomic64_dec_if_positive(atomic64_t *v);
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
|
||||
#endif /* !(__ARCH_SPARC64_ATOMIC__) */
|
||||
|
@ -95,7 +95,7 @@ int atomic_cmpxchg(atomic_t *v, int old, int new)
|
||||
}
|
||||
EXPORT_SYMBOL(atomic_cmpxchg);
|
||||
|
||||
int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int ret;
|
||||
unsigned long flags;
|
||||
@ -107,7 +107,7 @@ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
spin_unlock_irqrestore(ATOMIC_HASH(v), flags);
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(__atomic_add_unless);
|
||||
EXPORT_SYMBOL(atomic_fetch_add_unless);
|
||||
|
||||
/* Atomic operations are already serializing */
|
||||
void atomic_set(atomic_t *v, int i)
|
||||
|
@ -80,6 +80,7 @@ static __always_inline void arch_atomic_sub(int i, atomic_t *v)
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define arch_atomic_sub_and_test arch_atomic_sub_and_test
|
||||
static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
|
||||
{
|
||||
GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e);
|
||||
@ -91,6 +92,7 @@ static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define arch_atomic_inc arch_atomic_inc
|
||||
static __always_inline void arch_atomic_inc(atomic_t *v)
|
||||
{
|
||||
asm volatile(LOCK_PREFIX "incl %0"
|
||||
@ -103,6 +105,7 @@ static __always_inline void arch_atomic_inc(atomic_t *v)
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define arch_atomic_dec arch_atomic_dec
|
||||
static __always_inline void arch_atomic_dec(atomic_t *v)
|
||||
{
|
||||
asm volatile(LOCK_PREFIX "decl %0"
|
||||
@ -117,6 +120,7 @@ static __always_inline void arch_atomic_dec(atomic_t *v)
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#define arch_atomic_dec_and_test arch_atomic_dec_and_test
|
||||
static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
|
||||
{
|
||||
GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e);
|
||||
@ -130,6 +134,7 @@ static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define arch_atomic_inc_and_test arch_atomic_inc_and_test
|
||||
static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
|
||||
{
|
||||
GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e);
|
||||
@ -144,6 +149,7 @@ static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#define arch_atomic_add_negative arch_atomic_add_negative
|
||||
static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v)
|
||||
{
|
||||
GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s);
|
||||
@ -173,9 +179,6 @@ static __always_inline int arch_atomic_sub_return(int i, atomic_t *v)
|
||||
return arch_atomic_add_return(-i, v);
|
||||
}
|
||||
|
||||
#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v))
|
||||
#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v))
|
||||
|
||||
static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v)
|
||||
{
|
||||
return xadd(&v->counter, i);
|
||||
@ -199,7 +202,7 @@ static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int n
|
||||
|
||||
static inline int arch_atomic_xchg(atomic_t *v, int new)
|
||||
{
|
||||
return xchg(&v->counter, new);
|
||||
return arch_xchg(&v->counter, new);
|
||||
}
|
||||
|
||||
static inline void arch_atomic_and(int i, atomic_t *v)
|
||||
@ -253,27 +256,6 @@ static inline int arch_atomic_fetch_xor(int i, atomic_t *v)
|
||||
return val;
|
||||
}
|
||||
|
||||
/**
|
||||
* __arch_atomic_add_unless - add unless the number is already a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as @v was not already @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c = arch_atomic_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c == u))
|
||||
break;
|
||||
} while (!arch_atomic_try_cmpxchg(v, &c, c + a));
|
||||
|
||||
return c;
|
||||
}
|
||||
|
||||
#ifdef CONFIG_X86_32
|
||||
# include <asm/atomic64_32.h>
|
||||
#else
|
||||
|
@ -158,6 +158,7 @@ static inline long long arch_atomic64_inc_return(atomic64_t *v)
|
||||
"S" (v) : "memory", "ecx");
|
||||
return a;
|
||||
}
|
||||
#define arch_atomic64_inc_return arch_atomic64_inc_return
|
||||
|
||||
static inline long long arch_atomic64_dec_return(atomic64_t *v)
|
||||
{
|
||||
@ -166,6 +167,7 @@ static inline long long arch_atomic64_dec_return(atomic64_t *v)
|
||||
"S" (v) : "memory", "ecx");
|
||||
return a;
|
||||
}
|
||||
#define arch_atomic64_dec_return arch_atomic64_dec_return
|
||||
|
||||
/**
|
||||
* arch_atomic64_add - add integer to atomic64 variable
|
||||
@ -197,26 +199,13 @@ static inline long long arch_atomic64_sub(long long i, atomic64_t *v)
|
||||
return i;
|
||||
}
|
||||
|
||||
/**
|
||||
* arch_atomic64_sub_and_test - subtract value from variable and test result
|
||||
* @i: integer value to subtract
|
||||
* @v: pointer to type atomic64_t
|
||||
*
|
||||
* Atomically subtracts @i from @v and returns
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v)
|
||||
{
|
||||
return arch_atomic64_sub_return(i, v) == 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* arch_atomic64_inc - increment atomic64 variable
|
||||
* @v: pointer to type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define arch_atomic64_inc arch_atomic64_inc
|
||||
static inline void arch_atomic64_inc(atomic64_t *v)
|
||||
{
|
||||
__alternative_atomic64(inc, inc_return, /* no output */,
|
||||
@ -229,52 +218,13 @@ static inline void arch_atomic64_inc(atomic64_t *v)
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define arch_atomic64_dec arch_atomic64_dec
|
||||
static inline void arch_atomic64_dec(atomic64_t *v)
|
||||
{
|
||||
__alternative_atomic64(dec, dec_return, /* no output */,
|
||||
"S" (v) : "memory", "eax", "ecx", "edx");
|
||||
}
|
||||
|
||||
/**
|
||||
* arch_atomic64_dec_and_test - decrement and test
|
||||
* @v: pointer to type atomic64_t
|
||||
*
|
||||
* Atomically decrements @v by 1 and
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
static inline int arch_atomic64_dec_and_test(atomic64_t *v)
|
||||
{
|
||||
return arch_atomic64_dec_return(v) == 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* atomic64_inc_and_test - increment and test
|
||||
* @v: pointer to type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
static inline int arch_atomic64_inc_and_test(atomic64_t *v)
|
||||
{
|
||||
return arch_atomic64_inc_return(v) == 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* arch_atomic64_add_negative - add and test if negative
|
||||
* @i: integer value to add
|
||||
* @v: pointer to type atomic64_t
|
||||
*
|
||||
* Atomically adds @i to @v and returns true
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
static inline int arch_atomic64_add_negative(long long i, atomic64_t *v)
|
||||
{
|
||||
return arch_atomic64_add_return(i, v) < 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* arch_atomic64_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
@ -295,7 +245,7 @@ static inline int arch_atomic64_add_unless(atomic64_t *v, long long a,
|
||||
return (int)a;
|
||||
}
|
||||
|
||||
|
||||
#define arch_atomic64_inc_not_zero arch_atomic64_inc_not_zero
|
||||
static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
|
||||
{
|
||||
int r;
|
||||
@ -304,6 +254,7 @@ static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
|
||||
return r;
|
||||
}
|
||||
|
||||
#define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive
|
||||
static inline long long arch_atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
long long r;
|
||||
|
@ -71,6 +71,7 @@ static inline void arch_atomic64_sub(long i, atomic64_t *v)
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define arch_atomic64_sub_and_test arch_atomic64_sub_and_test
|
||||
static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
|
||||
{
|
||||
GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e);
|
||||
@ -82,6 +83,7 @@ static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define arch_atomic64_inc arch_atomic64_inc
|
||||
static __always_inline void arch_atomic64_inc(atomic64_t *v)
|
||||
{
|
||||
asm volatile(LOCK_PREFIX "incq %0"
|
||||
@ -95,6 +97,7 @@ static __always_inline void arch_atomic64_inc(atomic64_t *v)
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define arch_atomic64_dec arch_atomic64_dec
|
||||
static __always_inline void arch_atomic64_dec(atomic64_t *v)
|
||||
{
|
||||
asm volatile(LOCK_PREFIX "decq %0"
|
||||
@ -110,6 +113,7 @@ static __always_inline void arch_atomic64_dec(atomic64_t *v)
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#define arch_atomic64_dec_and_test arch_atomic64_dec_and_test
|
||||
static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
|
||||
{
|
||||
GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e);
|
||||
@ -123,6 +127,7 @@ static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define arch_atomic64_inc_and_test arch_atomic64_inc_and_test
|
||||
static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
|
||||
{
|
||||
GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e);
|
||||
@ -137,6 +142,7 @@ static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#define arch_atomic64_add_negative arch_atomic64_add_negative
|
||||
static inline bool arch_atomic64_add_negative(long i, atomic64_t *v)
|
||||
{
|
||||
GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s);
|
||||
@ -169,9 +175,6 @@ static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v)
|
||||
return xadd(&v->counter, -i);
|
||||
}
|
||||
|
||||
#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v)))
|
||||
#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v)))
|
||||
|
||||
static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new)
|
||||
{
|
||||
return arch_cmpxchg(&v->counter, old, new);
|
||||
@ -185,46 +188,7 @@ static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, l
|
||||
|
||||
static inline long arch_atomic64_xchg(atomic64_t *v, long new)
|
||||
{
|
||||
return xchg(&v->counter, new);
|
||||
}
|
||||
|
||||
/**
|
||||
* arch_atomic64_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u)
|
||||
{
|
||||
s64 c = arch_atomic64_read(v);
|
||||
do {
|
||||
if (unlikely(c == u))
|
||||
return false;
|
||||
} while (!arch_atomic64_try_cmpxchg(v, &c, c + a));
|
||||
return true;
|
||||
}
|
||||
|
||||
#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0)
|
||||
|
||||
/*
|
||||
* arch_atomic64_dec_if_positive - decrement by 1 if old value positive
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* The function returns the old value of *v minus 1, even if
|
||||
* the atomic variable, v, was not decremented.
|
||||
*/
|
||||
static inline long arch_atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
s64 dec, c = arch_atomic64_read(v);
|
||||
do {
|
||||
dec = c - 1;
|
||||
if (unlikely(dec < 0))
|
||||
break;
|
||||
} while (!arch_atomic64_try_cmpxchg(v, &c, dec));
|
||||
return dec;
|
||||
return arch_xchg(&v->counter, new);
|
||||
}
|
||||
|
||||
static inline void arch_atomic64_and(long i, atomic64_t *v)
|
||||
|
@ -75,7 +75,7 @@ extern void __add_wrong_size(void)
|
||||
* use "asm volatile" and "memory" clobbers to prevent gcc from moving
|
||||
* information around.
|
||||
*/
|
||||
#define xchg(ptr, v) __xchg_op((ptr), (v), xchg, "")
|
||||
#define arch_xchg(ptr, v) __xchg_op((ptr), (v), xchg, "")
|
||||
|
||||
/*
|
||||
* Atomic compare and exchange. Compare OLD with MEM, if identical,
|
||||
|
@ -10,13 +10,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val)
|
||||
#define arch_cmpxchg64(ptr, o, n) \
|
||||
({ \
|
||||
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
|
||||
cmpxchg((ptr), (o), (n)); \
|
||||
arch_cmpxchg((ptr), (o), (n)); \
|
||||
})
|
||||
|
||||
#define arch_cmpxchg64_local(ptr, o, n) \
|
||||
({ \
|
||||
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
|
||||
cmpxchg_local((ptr), (o), (n)); \
|
||||
arch_cmpxchg_local((ptr), (o), (n)); \
|
||||
})
|
||||
|
||||
#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
|
||||
|
@ -5,6 +5,7 @@
|
||||
* PaX/grsecurity.
|
||||
*/
|
||||
#include <linux/refcount.h>
|
||||
#include <asm/bug.h>
|
||||
|
||||
/*
|
||||
* This is the first portion of the refcount error handling, which lives in
|
||||
|
@ -197,107 +197,9 @@ ATOMIC_OPS(xor)
|
||||
#undef ATOMIC_OP_RETURN
|
||||
#undef ATOMIC_OP
|
||||
|
||||
/**
|
||||
* atomic_sub_and_test - subtract value from variable and test result
|
||||
* @i: integer value to subtract
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically subtracts @i from @v and returns
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_sub_and_test(i,v) (atomic_sub_return((i),(v)) == 0)
|
||||
|
||||
/**
|
||||
* atomic_inc - increment atomic variable
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define atomic_inc(v) atomic_add(1,(v))
|
||||
|
||||
/**
|
||||
* atomic_inc - increment atomic variable
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1.
|
||||
*/
|
||||
#define atomic_inc_return(v) atomic_add_return(1,(v))
|
||||
|
||||
/**
|
||||
* atomic_dec - decrement atomic variable
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define atomic_dec(v) atomic_sub(1,(v))
|
||||
|
||||
/**
|
||||
* atomic_dec_return - decrement atomic variable
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically decrements @v by 1.
|
||||
*/
|
||||
#define atomic_dec_return(v) atomic_sub_return(1,(v))
|
||||
|
||||
/**
|
||||
* atomic_dec_and_test - decrement and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically decrements @v by 1 and
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#define atomic_dec_and_test(v) (atomic_sub_return(1,(v)) == 0)
|
||||
|
||||
/**
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#define atomic_inc_and_test(v) (atomic_add_return(1,(v)) == 0)
|
||||
|
||||
/**
|
||||
* atomic_add_negative - add and test if negative
|
||||
* @v: pointer of type atomic_t
|
||||
* @i: integer value to add
|
||||
*
|
||||
* Atomically adds @i to @v and returns true
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#define atomic_add_negative(i,v) (atomic_add_return((i),(v)) < 0)
|
||||
|
||||
#define atomic_cmpxchg(v, o, n) ((int)cmpxchg(&((v)->counter), (o), (n)))
|
||||
#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
|
||||
|
||||
/**
|
||||
* __atomic_add_unless - add unless the number is a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as it was not @u.
|
||||
* Returns the old value of @v.
|
||||
*/
|
||||
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
if (unlikely(c == (u)))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, c + (a));
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
#endif /* __KERNEL__ */
|
||||
|
||||
#endif /* _XTENSA_ATOMIC_H */
|
||||
|
@ -61,7 +61,7 @@ static int atomic_inc_return_safe(atomic_t *v)
|
||||
{
|
||||
unsigned int counter;
|
||||
|
||||
counter = (unsigned int)__atomic_add_unless(v, 1, 0);
|
||||
counter = (unsigned int)atomic_fetch_add_unless(v, 1, 0);
|
||||
if (counter <= (unsigned int)INT_MAX)
|
||||
return (int)counter;
|
||||
|
||||
|
@ -121,7 +121,7 @@ static int uverbs_try_lock_object(struct ib_uobject *uobj, bool exclusive)
|
||||
* this lock.
|
||||
*/
|
||||
if (!exclusive)
|
||||
return __atomic_add_unless(&uobj->usecnt, 1, -1) == -1 ?
|
||||
return atomic_fetch_add_unless(&uobj->usecnt, 1, -1) == -1 ?
|
||||
-EBUSY : 0;
|
||||
|
||||
/* lock is either WRITE or DESTROY - should be exclusive */
|
||||
|
@ -648,7 +648,7 @@ static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall,
|
||||
trace_afs_notify_call(rxcall, call);
|
||||
call->need_attention = true;
|
||||
|
||||
u = __atomic_add_unless(&call->usage, 1, 0);
|
||||
u = atomic_fetch_add_unless(&call->usage, 1, 0);
|
||||
if (u != 0) {
|
||||
trace_afs_call(call, afs_call_trace_wake, u,
|
||||
atomic_read(&call->net->nr_outstanding_calls),
|
||||
|
@ -84,42 +84,59 @@ static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 ne
|
||||
}
|
||||
#endif
|
||||
|
||||
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
#ifdef arch_atomic_fetch_add_unless
|
||||
#define atomic_fetch_add_unless atomic_fetch_add_unless
|
||||
static __always_inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return __arch_atomic_add_unless(v, a, u);
|
||||
return arch_atomic_fetch_add_unless(v, a, u);
|
||||
}
|
||||
#endif
|
||||
|
||||
|
||||
static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
|
||||
#ifdef arch_atomic64_fetch_add_unless
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
static __always_inline s64 atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_add_unless(v, a, u);
|
||||
return arch_atomic64_fetch_add_unless(v, a, u);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic_inc
|
||||
#define atomic_inc atomic_inc
|
||||
static __always_inline void atomic_inc(atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
arch_atomic_inc(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_inc
|
||||
#define atomic64_inc atomic64_inc
|
||||
static __always_inline void atomic64_inc(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
arch_atomic64_inc(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic_dec
|
||||
#define atomic_dec atomic_dec
|
||||
static __always_inline void atomic_dec(atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
arch_atomic_dec(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef atch_atomic64_dec
|
||||
#define atomic64_dec
|
||||
static __always_inline void atomic64_dec(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
arch_atomic64_dec(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
static __always_inline void atomic_add(int i, atomic_t *v)
|
||||
{
|
||||
@ -181,65 +198,95 @@ static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
|
||||
arch_atomic64_xor(i, v);
|
||||
}
|
||||
|
||||
#ifdef arch_atomic_inc_return
|
||||
#define atomic_inc_return atomic_inc_return
|
||||
static __always_inline int atomic_inc_return(atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic_inc_return(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_in_return
|
||||
#define atomic64_inc_return atomic64_inc_return
|
||||
static __always_inline s64 atomic64_inc_return(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_inc_return(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic_dec_return
|
||||
#define atomic_dec_return atomic_dec_return
|
||||
static __always_inline int atomic_dec_return(atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic_dec_return(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_dec_return
|
||||
#define atomic64_dec_return atomic64_dec_return
|
||||
static __always_inline s64 atomic64_dec_return(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_dec_return(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
|
||||
#ifdef arch_atomic64_inc_not_zero
|
||||
#define atomic64_inc_not_zero atomic64_inc_not_zero
|
||||
static __always_inline bool atomic64_inc_not_zero(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_inc_not_zero(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_dec_if_positive
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_dec_if_positive(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic_dec_and_test
|
||||
#define atomic_dec_and_test atomic_dec_and_test
|
||||
static __always_inline bool atomic_dec_and_test(atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic_dec_and_test(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_dec_and_test
|
||||
#define atomic64_dec_and_test atomic64_dec_and_test
|
||||
static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_dec_and_test(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic_inc_and_test
|
||||
#define atomic_inc_and_test atomic_inc_and_test
|
||||
static __always_inline bool atomic_inc_and_test(atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic_inc_and_test(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_inc_and_test
|
||||
#define atomic64_inc_and_test atomic64_inc_and_test
|
||||
static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_inc_and_test(v);
|
||||
}
|
||||
#endif
|
||||
|
||||
static __always_inline int atomic_add_return(int i, atomic_t *v)
|
||||
{
|
||||
@ -325,152 +372,96 @@ static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
|
||||
return arch_atomic64_fetch_xor(i, v);
|
||||
}
|
||||
|
||||
#ifdef arch_atomic_sub_and_test
|
||||
#define atomic_sub_and_test atomic_sub_and_test
|
||||
static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic_sub_and_test(i, v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_sub_and_test
|
||||
#define atomic64_sub_and_test atomic64_sub_and_test
|
||||
static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_sub_and_test(i, v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic_add_negative
|
||||
#define atomic_add_negative atomic_add_negative
|
||||
static __always_inline bool atomic_add_negative(int i, atomic_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic_add_negative(i, v);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef arch_atomic64_add_negative
|
||||
#define atomic64_add_negative atomic64_add_negative
|
||||
static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
|
||||
{
|
||||
kasan_check_write(v, sizeof(*v));
|
||||
return arch_atomic64_add_negative(i, v);
|
||||
}
|
||||
#endif
|
||||
|
||||
static __always_inline unsigned long
|
||||
cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
|
||||
{
|
||||
kasan_check_write(ptr, size);
|
||||
switch (size) {
|
||||
case 1:
|
||||
return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
|
||||
case 2:
|
||||
return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
|
||||
case 4:
|
||||
return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
|
||||
case 8:
|
||||
BUILD_BUG_ON(sizeof(unsigned long) != 8);
|
||||
return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
|
||||
}
|
||||
BUILD_BUG();
|
||||
return 0;
|
||||
}
|
||||
#define xchg(ptr, new) \
|
||||
({ \
|
||||
typeof(ptr) __ai_ptr = (ptr); \
|
||||
kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
|
||||
arch_xchg(__ai_ptr, (new)); \
|
||||
})
|
||||
|
||||
#define cmpxchg(ptr, old, new) \
|
||||
({ \
|
||||
((__typeof__(*(ptr)))cmpxchg_size((ptr), (unsigned long)(old), \
|
||||
(unsigned long)(new), sizeof(*(ptr)))); \
|
||||
typeof(ptr) __ai_ptr = (ptr); \
|
||||
kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
|
||||
arch_cmpxchg(__ai_ptr, (old), (new)); \
|
||||
})
|
||||
|
||||
static __always_inline unsigned long
|
||||
sync_cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new,
|
||||
int size)
|
||||
{
|
||||
kasan_check_write(ptr, size);
|
||||
switch (size) {
|
||||
case 1:
|
||||
return arch_sync_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
|
||||
case 2:
|
||||
return arch_sync_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
|
||||
case 4:
|
||||
return arch_sync_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
|
||||
case 8:
|
||||
BUILD_BUG_ON(sizeof(unsigned long) != 8);
|
||||
return arch_sync_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
|
||||
}
|
||||
BUILD_BUG();
|
||||
return 0;
|
||||
}
|
||||
|
||||
#define sync_cmpxchg(ptr, old, new) \
|
||||
({ \
|
||||
((__typeof__(*(ptr)))sync_cmpxchg_size((ptr), \
|
||||
(unsigned long)(old), (unsigned long)(new), \
|
||||
sizeof(*(ptr)))); \
|
||||
typeof(ptr) __ai_ptr = (ptr); \
|
||||
kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
|
||||
arch_sync_cmpxchg(__ai_ptr, (old), (new)); \
|
||||
})
|
||||
|
||||
static __always_inline unsigned long
|
||||
cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
|
||||
int size)
|
||||
{
|
||||
kasan_check_write(ptr, size);
|
||||
switch (size) {
|
||||
case 1:
|
||||
return arch_cmpxchg_local((u8 *)ptr, (u8)old, (u8)new);
|
||||
case 2:
|
||||
return arch_cmpxchg_local((u16 *)ptr, (u16)old, (u16)new);
|
||||
case 4:
|
||||
return arch_cmpxchg_local((u32 *)ptr, (u32)old, (u32)new);
|
||||
case 8:
|
||||
BUILD_BUG_ON(sizeof(unsigned long) != 8);
|
||||
return arch_cmpxchg_local((u64 *)ptr, (u64)old, (u64)new);
|
||||
}
|
||||
BUILD_BUG();
|
||||
return 0;
|
||||
}
|
||||
|
||||
#define cmpxchg_local(ptr, old, new) \
|
||||
({ \
|
||||
((__typeof__(*(ptr)))cmpxchg_local_size((ptr), \
|
||||
(unsigned long)(old), (unsigned long)(new), \
|
||||
sizeof(*(ptr)))); \
|
||||
typeof(ptr) __ai_ptr = (ptr); \
|
||||
kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
|
||||
arch_cmpxchg_local(__ai_ptr, (old), (new)); \
|
||||
})
|
||||
|
||||
static __always_inline u64
|
||||
cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
|
||||
{
|
||||
kasan_check_write(ptr, sizeof(*ptr));
|
||||
return arch_cmpxchg64(ptr, old, new);
|
||||
}
|
||||
|
||||
#define cmpxchg64(ptr, old, new) \
|
||||
({ \
|
||||
((__typeof__(*(ptr)))cmpxchg64_size((ptr), (u64)(old), \
|
||||
(u64)(new))); \
|
||||
typeof(ptr) __ai_ptr = (ptr); \
|
||||
kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
|
||||
arch_cmpxchg64(__ai_ptr, (old), (new)); \
|
||||
})
|
||||
|
||||
static __always_inline u64
|
||||
cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
|
||||
{
|
||||
kasan_check_write(ptr, sizeof(*ptr));
|
||||
return arch_cmpxchg64_local(ptr, old, new);
|
||||
}
|
||||
|
||||
#define cmpxchg64_local(ptr, old, new) \
|
||||
({ \
|
||||
((__typeof__(*(ptr)))cmpxchg64_local_size((ptr), (u64)(old), \
|
||||
(u64)(new))); \
|
||||
typeof(ptr) __ai_ptr = (ptr); \
|
||||
kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
|
||||
arch_cmpxchg64_local(__ai_ptr, (old), (new)); \
|
||||
})
|
||||
|
||||
/*
|
||||
* Originally we had the following code here:
|
||||
* __typeof__(p1) ____p1 = (p1);
|
||||
* kasan_check_write(____p1, 2 * sizeof(*____p1));
|
||||
* arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2));
|
||||
* But it leads to compilation failures (see gcc issue 72873).
|
||||
* So for now it's left non-instrumented.
|
||||
* There are few callers of cmpxchg_double(), so it's not critical.
|
||||
*/
|
||||
#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
|
||||
({ \
|
||||
arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
|
||||
typeof(p1) __ai_p1 = (p1); \
|
||||
kasan_check_write(__ai_p1, 2 * sizeof(*__ai_p1)); \
|
||||
arch_cmpxchg_double(__ai_p1, (p2), (o1), (o2), (n1), (n2)); \
|
||||
})
|
||||
|
||||
#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
|
||||
({ \
|
||||
arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \
|
||||
#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
|
||||
({ \
|
||||
typeof(p1) __ai_p1 = (p1); \
|
||||
kasan_check_write(__ai_p1, 2 * sizeof(*__ai_p1)); \
|
||||
arch_cmpxchg_double_local(__ai_p1, (p2), (o1), (o2), (n1), (n2)); \
|
||||
})
|
||||
|
||||
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
|
||||
|
@ -186,11 +186,6 @@ ATOMIC_OP(xor, ^)
|
||||
|
||||
#include <linux/irqflags.h>
|
||||
|
||||
static inline int atomic_add_negative(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_add_return(i, v) < 0;
|
||||
}
|
||||
|
||||
static inline void atomic_add(int i, atomic_t *v)
|
||||
{
|
||||
atomic_add_return(i, v);
|
||||
@ -201,35 +196,7 @@ static inline void atomic_sub(int i, atomic_t *v)
|
||||
atomic_sub_return(i, v);
|
||||
}
|
||||
|
||||
static inline void atomic_inc(atomic_t *v)
|
||||
{
|
||||
atomic_add_return(1, v);
|
||||
}
|
||||
|
||||
static inline void atomic_dec(atomic_t *v)
|
||||
{
|
||||
atomic_sub_return(1, v);
|
||||
}
|
||||
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
|
||||
#define atomic_sub_and_test(i, v) (atomic_sub_return((i), (v)) == 0)
|
||||
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
|
||||
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
|
||||
|
||||
#define atomic_xchg(ptr, v) (xchg(&(ptr)->counter, (v)))
|
||||
#define atomic_cmpxchg(v, old, new) (cmpxchg(&((v)->counter), (old), (new)))
|
||||
|
||||
#ifndef __atomic_add_unless
|
||||
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c, old;
|
||||
c = atomic_read(v);
|
||||
while (c != u && (old = atomic_cmpxchg(v, c, c + a)) != c)
|
||||
c = old;
|
||||
return c;
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* __ASM_GENERIC_ATOMIC_H */
|
||||
|
@ -11,6 +11,7 @@
|
||||
*/
|
||||
#ifndef _ASM_GENERIC_ATOMIC64_H
|
||||
#define _ASM_GENERIC_ATOMIC64_H
|
||||
#include <linux/types.h>
|
||||
|
||||
typedef struct {
|
||||
long long counter;
|
||||
@ -50,18 +51,10 @@ ATOMIC64_OPS(xor)
|
||||
#undef ATOMIC64_OP
|
||||
|
||||
extern long long atomic64_dec_if_positive(atomic64_t *v);
|
||||
#define atomic64_dec_if_positive atomic64_dec_if_positive
|
||||
extern long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n);
|
||||
extern long long atomic64_xchg(atomic64_t *v, long long new);
|
||||
extern int atomic64_add_unless(atomic64_t *v, long long a, long long u);
|
||||
|
||||
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
|
||||
#define atomic64_inc(v) atomic64_add(1LL, (v))
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1LL, (v))
|
||||
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
|
||||
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
|
||||
#define atomic64_dec(v) atomic64_sub(1LL, (v))
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1LL, (v))
|
||||
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1LL, 0LL)
|
||||
extern long long atomic64_fetch_add_unless(atomic64_t *v, long long a, long long u);
|
||||
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
|
||||
|
||||
#endif /* _ASM_GENERIC_ATOMIC64_H */
|
||||
|
@ -2,189 +2,67 @@
|
||||
#ifndef _ASM_GENERIC_BITOPS_ATOMIC_H_
|
||||
#define _ASM_GENERIC_BITOPS_ATOMIC_H_
|
||||
|
||||
#include <asm/types.h>
|
||||
#include <linux/irqflags.h>
|
||||
|
||||
#ifdef CONFIG_SMP
|
||||
#include <asm/spinlock.h>
|
||||
#include <asm/cache.h> /* we use L1_CACHE_BYTES */
|
||||
|
||||
/* Use an array of spinlocks for our atomic_ts.
|
||||
* Hash function to index into a different SPINLOCK.
|
||||
* Since "a" is usually an address, use one spinlock per cacheline.
|
||||
*/
|
||||
# define ATOMIC_HASH_SIZE 4
|
||||
# define ATOMIC_HASH(a) (&(__atomic_hash[ (((unsigned long) a)/L1_CACHE_BYTES) & (ATOMIC_HASH_SIZE-1) ]))
|
||||
|
||||
extern arch_spinlock_t __atomic_hash[ATOMIC_HASH_SIZE] __lock_aligned;
|
||||
|
||||
/* Can't use raw_spin_lock_irq because of #include problems, so
|
||||
* this is the substitute */
|
||||
#define _atomic_spin_lock_irqsave(l,f) do { \
|
||||
arch_spinlock_t *s = ATOMIC_HASH(l); \
|
||||
local_irq_save(f); \
|
||||
arch_spin_lock(s); \
|
||||
} while(0)
|
||||
|
||||
#define _atomic_spin_unlock_irqrestore(l,f) do { \
|
||||
arch_spinlock_t *s = ATOMIC_HASH(l); \
|
||||
arch_spin_unlock(s); \
|
||||
local_irq_restore(f); \
|
||||
} while(0)
|
||||
|
||||
|
||||
#else
|
||||
# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save(f); } while (0)
|
||||
# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore(f); } while (0)
|
||||
#endif
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <asm/barrier.h>
|
||||
|
||||
/*
|
||||
* NMI events can occur at any time, including when interrupts have been
|
||||
* disabled by *_irqsave(). So you can get NMI events occurring while a
|
||||
* *_bit function is holding a spin lock. If the NMI handler also wants
|
||||
* to do bit manipulation (and they do) then you can get a deadlock
|
||||
* between the original caller of *_bit() and the NMI handler.
|
||||
*
|
||||
* by Keith Owens
|
||||
* Implementation of atomic bitops using atomic-fetch ops.
|
||||
* See Documentation/atomic_bitops.txt for details.
|
||||
*/
|
||||
|
||||
/**
|
||||
* set_bit - Atomically set a bit in memory
|
||||
* @nr: the bit to set
|
||||
* @addr: the address to start counting from
|
||||
*
|
||||
* This function is atomic and may not be reordered. See __set_bit()
|
||||
* if you do not require the atomic guarantees.
|
||||
*
|
||||
* Note: there are no guarantees that this function will not be reordered
|
||||
* on non x86 architectures, so if you are writing portable code,
|
||||
* make sure not to rely on its reordering guarantees.
|
||||
*
|
||||
* Note that @nr may be almost arbitrarily large; this function is not
|
||||
* restricted to acting on a single-word quantity.
|
||||
*/
|
||||
static inline void set_bit(int nr, volatile unsigned long *addr)
|
||||
static inline void set_bit(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
|
||||
unsigned long flags;
|
||||
|
||||
_atomic_spin_lock_irqsave(p, flags);
|
||||
*p |= mask;
|
||||
_atomic_spin_unlock_irqrestore(p, flags);
|
||||
p += BIT_WORD(nr);
|
||||
atomic_long_or(BIT_MASK(nr), (atomic_long_t *)p);
|
||||
}
|
||||
|
||||
/**
|
||||
* clear_bit - Clears a bit in memory
|
||||
* @nr: Bit to clear
|
||||
* @addr: Address to start counting from
|
||||
*
|
||||
* clear_bit() is atomic and may not be reordered. However, it does
|
||||
* not contain a memory barrier, so if it is used for locking purposes,
|
||||
* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic()
|
||||
* in order to ensure changes are visible on other processors.
|
||||
*/
|
||||
static inline void clear_bit(int nr, volatile unsigned long *addr)
|
||||
static inline void clear_bit(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
|
||||
unsigned long flags;
|
||||
|
||||
_atomic_spin_lock_irqsave(p, flags);
|
||||
*p &= ~mask;
|
||||
_atomic_spin_unlock_irqrestore(p, flags);
|
||||
p += BIT_WORD(nr);
|
||||
atomic_long_andnot(BIT_MASK(nr), (atomic_long_t *)p);
|
||||
}
|
||||
|
||||
/**
|
||||
* change_bit - Toggle a bit in memory
|
||||
* @nr: Bit to change
|
||||
* @addr: Address to start counting from
|
||||
*
|
||||
* change_bit() is atomic and may not be reordered. It may be
|
||||
* reordered on other architectures than x86.
|
||||
* Note that @nr may be almost arbitrarily large; this function is not
|
||||
* restricted to acting on a single-word quantity.
|
||||
*/
|
||||
static inline void change_bit(int nr, volatile unsigned long *addr)
|
||||
static inline void change_bit(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
|
||||
unsigned long flags;
|
||||
|
||||
_atomic_spin_lock_irqsave(p, flags);
|
||||
*p ^= mask;
|
||||
_atomic_spin_unlock_irqrestore(p, flags);
|
||||
p += BIT_WORD(nr);
|
||||
atomic_long_xor(BIT_MASK(nr), (atomic_long_t *)p);
|
||||
}
|
||||
|
||||
/**
|
||||
* test_and_set_bit - Set a bit and return its old value
|
||||
* @nr: Bit to set
|
||||
* @addr: Address to count from
|
||||
*
|
||||
* This operation is atomic and cannot be reordered.
|
||||
* It may be reordered on other architectures than x86.
|
||||
* It also implies a memory barrier.
|
||||
*/
|
||||
static inline int test_and_set_bit(int nr, volatile unsigned long *addr)
|
||||
static inline int test_and_set_bit(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
long old;
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
|
||||
unsigned long old;
|
||||
unsigned long flags;
|
||||
|
||||
_atomic_spin_lock_irqsave(p, flags);
|
||||
old = *p;
|
||||
*p = old | mask;
|
||||
_atomic_spin_unlock_irqrestore(p, flags);
|
||||
p += BIT_WORD(nr);
|
||||
if (READ_ONCE(*p) & mask)
|
||||
return 1;
|
||||
|
||||
return (old & mask) != 0;
|
||||
old = atomic_long_fetch_or(mask, (atomic_long_t *)p);
|
||||
return !!(old & mask);
|
||||
}
|
||||
|
||||
/**
|
||||
* test_and_clear_bit - Clear a bit and return its old value
|
||||
* @nr: Bit to clear
|
||||
* @addr: Address to count from
|
||||
*
|
||||
* This operation is atomic and cannot be reordered.
|
||||
* It can be reorderdered on other architectures other than x86.
|
||||
* It also implies a memory barrier.
|
||||
*/
|
||||
static inline int test_and_clear_bit(int nr, volatile unsigned long *addr)
|
||||
static inline int test_and_clear_bit(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
long old;
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
|
||||
unsigned long old;
|
||||
unsigned long flags;
|
||||
|
||||
_atomic_spin_lock_irqsave(p, flags);
|
||||
old = *p;
|
||||
*p = old & ~mask;
|
||||
_atomic_spin_unlock_irqrestore(p, flags);
|
||||
p += BIT_WORD(nr);
|
||||
if (!(READ_ONCE(*p) & mask))
|
||||
return 0;
|
||||
|
||||
return (old & mask) != 0;
|
||||
old = atomic_long_fetch_andnot(mask, (atomic_long_t *)p);
|
||||
return !!(old & mask);
|
||||
}
|
||||
|
||||
/**
|
||||
* test_and_change_bit - Change a bit and return its old value
|
||||
* @nr: Bit to change
|
||||
* @addr: Address to count from
|
||||
*
|
||||
* This operation is atomic and cannot be reordered.
|
||||
* It also implies a memory barrier.
|
||||
*/
|
||||
static inline int test_and_change_bit(int nr, volatile unsigned long *addr)
|
||||
static inline int test_and_change_bit(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
long old;
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
|
||||
unsigned long old;
|
||||
unsigned long flags;
|
||||
|
||||
_atomic_spin_lock_irqsave(p, flags);
|
||||
old = *p;
|
||||
*p = old ^ mask;
|
||||
_atomic_spin_unlock_irqrestore(p, flags);
|
||||
|
||||
return (old & mask) != 0;
|
||||
p += BIT_WORD(nr);
|
||||
old = atomic_long_fetch_xor(mask, (atomic_long_t *)p);
|
||||
return !!(old & mask);
|
||||
}
|
||||
|
||||
#endif /* _ASM_GENERIC_BITOPS_ATOMIC_H */
|
||||
|
@ -2,6 +2,10 @@
|
||||
#ifndef _ASM_GENERIC_BITOPS_LOCK_H_
|
||||
#define _ASM_GENERIC_BITOPS_LOCK_H_
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <asm/barrier.h>
|
||||
|
||||
/**
|
||||
* test_and_set_bit_lock - Set a bit and return its old value, for lock
|
||||
* @nr: Bit to set
|
||||
@ -11,7 +15,20 @@
|
||||
* the returned value is 0.
|
||||
* It can be used to implement bit locks.
|
||||
*/
|
||||
#define test_and_set_bit_lock(nr, addr) test_and_set_bit(nr, addr)
|
||||
static inline int test_and_set_bit_lock(unsigned int nr,
|
||||
volatile unsigned long *p)
|
||||
{
|
||||
long old;
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
|
||||
p += BIT_WORD(nr);
|
||||
if (READ_ONCE(*p) & mask)
|
||||
return 1;
|
||||
|
||||
old = atomic_long_fetch_or_acquire(mask, (atomic_long_t *)p);
|
||||
return !!(old & mask);
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* clear_bit_unlock - Clear a bit in memory, for unlock
|
||||
@ -20,11 +37,11 @@
|
||||
*
|
||||
* This operation is atomic and provides release barrier semantics.
|
||||
*/
|
||||
#define clear_bit_unlock(nr, addr) \
|
||||
do { \
|
||||
smp_mb__before_atomic(); \
|
||||
clear_bit(nr, addr); \
|
||||
} while (0)
|
||||
static inline void clear_bit_unlock(unsigned int nr, volatile unsigned long *p)
|
||||
{
|
||||
p += BIT_WORD(nr);
|
||||
atomic_long_fetch_andnot_release(BIT_MASK(nr), (atomic_long_t *)p);
|
||||
}
|
||||
|
||||
/**
|
||||
* __clear_bit_unlock - Clear a bit in memory, for unlock
|
||||
@ -37,11 +54,38 @@ do { \
|
||||
*
|
||||
* See for example x86's implementation.
|
||||
*/
|
||||
#define __clear_bit_unlock(nr, addr) \
|
||||
do { \
|
||||
smp_mb__before_atomic(); \
|
||||
clear_bit(nr, addr); \
|
||||
} while (0)
|
||||
static inline void __clear_bit_unlock(unsigned int nr,
|
||||
volatile unsigned long *p)
|
||||
{
|
||||
unsigned long old;
|
||||
|
||||
p += BIT_WORD(nr);
|
||||
old = READ_ONCE(*p);
|
||||
old &= ~BIT_MASK(nr);
|
||||
atomic_long_set_release((atomic_long_t *)p, old);
|
||||
}
|
||||
|
||||
/**
|
||||
* clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom
|
||||
* byte is negative, for unlock.
|
||||
* @nr: the bit to clear
|
||||
* @addr: the address to start counting from
|
||||
*
|
||||
* This is a bit of a one-trick-pony for the filemap code, which clears
|
||||
* PG_locked and tests PG_waiters,
|
||||
*/
|
||||
#ifndef clear_bit_unlock_is_negative_byte
|
||||
static inline bool clear_bit_unlock_is_negative_byte(unsigned int nr,
|
||||
volatile unsigned long *p)
|
||||
{
|
||||
long old;
|
||||
unsigned long mask = BIT_MASK(nr);
|
||||
|
||||
p += BIT_WORD(nr);
|
||||
old = atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
|
||||
return !!(old & BIT(7));
|
||||
}
|
||||
#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte
|
||||
#endif
|
||||
|
||||
#endif /* _ASM_GENERIC_BITOPS_LOCK_H_ */
|
||||
|
||||
|
@ -2,6 +2,8 @@
|
||||
/* Atomic operations usable in machine independent code */
|
||||
#ifndef _LINUX_ATOMIC_H
|
||||
#define _LINUX_ATOMIC_H
|
||||
#include <linux/types.h>
|
||||
|
||||
#include <asm/atomic.h>
|
||||
#include <asm/barrier.h>
|
||||
|
||||
@ -36,40 +38,46 @@
|
||||
* barriers on top of the relaxed variant. In the case where the relaxed
|
||||
* variant is already fully ordered, no additional barriers are needed.
|
||||
*
|
||||
* Besides, if an arch has a special barrier for acquire/release, it could
|
||||
* implement its own __atomic_op_* and use the same framework for building
|
||||
* variants
|
||||
*
|
||||
* If an architecture overrides __atomic_op_acquire() it will probably want
|
||||
* to define smp_mb__after_spinlock().
|
||||
* If an architecture overrides __atomic_acquire_fence() it will probably
|
||||
* want to define smp_mb__after_spinlock().
|
||||
*/
|
||||
#ifndef __atomic_op_acquire
|
||||
#ifndef __atomic_acquire_fence
|
||||
#define __atomic_acquire_fence smp_mb__after_atomic
|
||||
#endif
|
||||
|
||||
#ifndef __atomic_release_fence
|
||||
#define __atomic_release_fence smp_mb__before_atomic
|
||||
#endif
|
||||
|
||||
#ifndef __atomic_pre_full_fence
|
||||
#define __atomic_pre_full_fence smp_mb__before_atomic
|
||||
#endif
|
||||
|
||||
#ifndef __atomic_post_full_fence
|
||||
#define __atomic_post_full_fence smp_mb__after_atomic
|
||||
#endif
|
||||
|
||||
#define __atomic_op_acquire(op, args...) \
|
||||
({ \
|
||||
typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \
|
||||
smp_mb__after_atomic(); \
|
||||
__atomic_acquire_fence(); \
|
||||
__ret; \
|
||||
})
|
||||
#endif
|
||||
|
||||
#ifndef __atomic_op_release
|
||||
#define __atomic_op_release(op, args...) \
|
||||
({ \
|
||||
smp_mb__before_atomic(); \
|
||||
__atomic_release_fence(); \
|
||||
op##_relaxed(args); \
|
||||
})
|
||||
#endif
|
||||
|
||||
#ifndef __atomic_op_fence
|
||||
#define __atomic_op_fence(op, args...) \
|
||||
({ \
|
||||
typeof(op##_relaxed(args)) __ret; \
|
||||
smp_mb__before_atomic(); \
|
||||
__atomic_pre_full_fence(); \
|
||||
__ret = op##_relaxed(args); \
|
||||
smp_mb__after_atomic(); \
|
||||
__atomic_post_full_fence(); \
|
||||
__ret; \
|
||||
})
|
||||
#endif
|
||||
|
||||
/* atomic_add_return_relaxed */
|
||||
#ifndef atomic_add_return_relaxed
|
||||
@ -95,11 +103,23 @@
|
||||
#endif
|
||||
#endif /* atomic_add_return_relaxed */
|
||||
|
||||
#ifndef atomic_inc
|
||||
#define atomic_inc(v) atomic_add(1, (v))
|
||||
#endif
|
||||
|
||||
/* atomic_inc_return_relaxed */
|
||||
#ifndef atomic_inc_return_relaxed
|
||||
|
||||
#ifndef atomic_inc_return
|
||||
#define atomic_inc_return(v) atomic_add_return(1, (v))
|
||||
#define atomic_inc_return_relaxed(v) atomic_add_return_relaxed(1, (v))
|
||||
#define atomic_inc_return_acquire(v) atomic_add_return_acquire(1, (v))
|
||||
#define atomic_inc_return_release(v) atomic_add_return_release(1, (v))
|
||||
#else /* atomic_inc_return */
|
||||
#define atomic_inc_return_relaxed atomic_inc_return
|
||||
#define atomic_inc_return_acquire atomic_inc_return
|
||||
#define atomic_inc_return_release atomic_inc_return
|
||||
#endif /* atomic_inc_return */
|
||||
|
||||
#else /* atomic_inc_return_relaxed */
|
||||
|
||||
@ -143,11 +163,23 @@
|
||||
#endif
|
||||
#endif /* atomic_sub_return_relaxed */
|
||||
|
||||
#ifndef atomic_dec
|
||||
#define atomic_dec(v) atomic_sub(1, (v))
|
||||
#endif
|
||||
|
||||
/* atomic_dec_return_relaxed */
|
||||
#ifndef atomic_dec_return_relaxed
|
||||
|
||||
#ifndef atomic_dec_return
|
||||
#define atomic_dec_return(v) atomic_sub_return(1, (v))
|
||||
#define atomic_dec_return_relaxed(v) atomic_sub_return_relaxed(1, (v))
|
||||
#define atomic_dec_return_acquire(v) atomic_sub_return_acquire(1, (v))
|
||||
#define atomic_dec_return_release(v) atomic_sub_return_release(1, (v))
|
||||
#else /* atomic_dec_return */
|
||||
#define atomic_dec_return_relaxed atomic_dec_return
|
||||
#define atomic_dec_return_acquire atomic_dec_return
|
||||
#define atomic_dec_return_release atomic_dec_return
|
||||
#endif /* atomic_dec_return */
|
||||
|
||||
#else /* atomic_dec_return_relaxed */
|
||||
|
||||
@ -328,12 +360,22 @@
|
||||
#endif
|
||||
#endif /* atomic_fetch_and_relaxed */
|
||||
|
||||
#ifdef atomic_andnot
|
||||
/* atomic_fetch_andnot_relaxed */
|
||||
#ifndef atomic_andnot
|
||||
#define atomic_andnot(i, v) atomic_and(~(int)(i), (v))
|
||||
#endif
|
||||
|
||||
#ifndef atomic_fetch_andnot_relaxed
|
||||
#define atomic_fetch_andnot_relaxed atomic_fetch_andnot
|
||||
#define atomic_fetch_andnot_acquire atomic_fetch_andnot
|
||||
#define atomic_fetch_andnot_release atomic_fetch_andnot
|
||||
|
||||
#ifndef atomic_fetch_andnot
|
||||
#define atomic_fetch_andnot(i, v) atomic_fetch_and(~(int)(i), (v))
|
||||
#define atomic_fetch_andnot_relaxed(i, v) atomic_fetch_and_relaxed(~(int)(i), (v))
|
||||
#define atomic_fetch_andnot_acquire(i, v) atomic_fetch_and_acquire(~(int)(i), (v))
|
||||
#define atomic_fetch_andnot_release(i, v) atomic_fetch_and_release(~(int)(i), (v))
|
||||
#else /* atomic_fetch_andnot */
|
||||
#define atomic_fetch_andnot_relaxed atomic_fetch_andnot
|
||||
#define atomic_fetch_andnot_acquire atomic_fetch_andnot
|
||||
#define atomic_fetch_andnot_release atomic_fetch_andnot
|
||||
#endif /* atomic_fetch_andnot */
|
||||
|
||||
#else /* atomic_fetch_andnot_relaxed */
|
||||
|
||||
@ -352,7 +394,6 @@
|
||||
__atomic_op_fence(atomic_fetch_andnot, __VA_ARGS__)
|
||||
#endif
|
||||
#endif /* atomic_fetch_andnot_relaxed */
|
||||
#endif /* atomic_andnot */
|
||||
|
||||
/* atomic_fetch_xor_relaxed */
|
||||
#ifndef atomic_fetch_xor_relaxed
|
||||
@ -519,113 +560,141 @@
|
||||
#endif
|
||||
#endif /* xchg_relaxed */
|
||||
|
||||
/**
|
||||
* atomic_fetch_add_unless - add unless the number is already a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, if @v was not already @u.
|
||||
* Returns the original value of @v.
|
||||
*/
|
||||
#ifndef atomic_fetch_add_unless
|
||||
static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
int c = atomic_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c == u))
|
||||
break;
|
||||
} while (!atomic_try_cmpxchg(v, &c, c + a));
|
||||
|
||||
return c;
|
||||
}
|
||||
#endif
|
||||
|
||||
/**
|
||||
* atomic_add_unless - add unless the number is already a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, so long as @v was not already @u.
|
||||
* Returns non-zero if @v was not @u, and zero otherwise.
|
||||
* Atomically adds @a to @v, if @v was not already @u.
|
||||
* Returns true if the addition was done.
|
||||
*/
|
||||
static inline int atomic_add_unless(atomic_t *v, int a, int u)
|
||||
static inline bool atomic_add_unless(atomic_t *v, int a, int u)
|
||||
{
|
||||
return __atomic_add_unless(v, a, u) != u;
|
||||
return atomic_fetch_add_unless(v, a, u) != u;
|
||||
}
|
||||
|
||||
/**
|
||||
* atomic_inc_not_zero - increment unless the number is zero
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1, so long as @v is non-zero.
|
||||
* Returns non-zero if @v was non-zero, and zero otherwise.
|
||||
* Atomically increments @v by 1, if @v is non-zero.
|
||||
* Returns true if the increment was done.
|
||||
*/
|
||||
#ifndef atomic_inc_not_zero
|
||||
#define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)
|
||||
#endif
|
||||
|
||||
#ifndef atomic_andnot
|
||||
static inline void atomic_andnot(int i, atomic_t *v)
|
||||
/**
|
||||
* atomic_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#ifndef atomic_inc_and_test
|
||||
static inline bool atomic_inc_and_test(atomic_t *v)
|
||||
{
|
||||
atomic_and(~i, v);
|
||||
}
|
||||
|
||||
static inline int atomic_fetch_andnot(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_fetch_and(~i, v);
|
||||
}
|
||||
|
||||
static inline int atomic_fetch_andnot_relaxed(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_fetch_and_relaxed(~i, v);
|
||||
}
|
||||
|
||||
static inline int atomic_fetch_andnot_acquire(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_fetch_and_acquire(~i, v);
|
||||
}
|
||||
|
||||
static inline int atomic_fetch_andnot_release(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_fetch_and_release(~i, v);
|
||||
return atomic_inc_return(v) == 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
/**
|
||||
* atomic_inc_not_zero_hint - increment if not null
|
||||
* atomic_dec_and_test - decrement and test
|
||||
* @v: pointer of type atomic_t
|
||||
* @hint: probable value of the atomic before the increment
|
||||
*
|
||||
* This version of atomic_inc_not_zero() gives a hint of probable
|
||||
* value of the atomic. This helps processor to not read the memory
|
||||
* before doing the atomic read/modify/write cycle, lowering
|
||||
* number of bus transactions on some arches.
|
||||
*
|
||||
* Returns: 0 if increment was not done, 1 otherwise.
|
||||
* Atomically decrements @v by 1 and
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#ifndef atomic_inc_not_zero_hint
|
||||
static inline int atomic_inc_not_zero_hint(atomic_t *v, int hint)
|
||||
#ifndef atomic_dec_and_test
|
||||
static inline bool atomic_dec_and_test(atomic_t *v)
|
||||
{
|
||||
int val, c = hint;
|
||||
return atomic_dec_return(v) == 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
/* sanity test, should be removed by compiler if hint is a constant */
|
||||
if (!hint)
|
||||
return atomic_inc_not_zero(v);
|
||||
/**
|
||||
* atomic_sub_and_test - subtract value from variable and test result
|
||||
* @i: integer value to subtract
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically subtracts @i from @v and returns
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#ifndef atomic_sub_and_test
|
||||
static inline bool atomic_sub_and_test(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_sub_return(i, v) == 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
do {
|
||||
val = atomic_cmpxchg(v, c, c + 1);
|
||||
if (val == c)
|
||||
return 1;
|
||||
c = val;
|
||||
} while (c);
|
||||
|
||||
return 0;
|
||||
/**
|
||||
* atomic_add_negative - add and test if negative
|
||||
* @i: integer value to add
|
||||
* @v: pointer of type atomic_t
|
||||
*
|
||||
* Atomically adds @i to @v and returns true
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#ifndef atomic_add_negative
|
||||
static inline bool atomic_add_negative(int i, atomic_t *v)
|
||||
{
|
||||
return atomic_add_return(i, v) < 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifndef atomic_inc_unless_negative
|
||||
static inline int atomic_inc_unless_negative(atomic_t *p)
|
||||
static inline bool atomic_inc_unless_negative(atomic_t *v)
|
||||
{
|
||||
int v, v1;
|
||||
for (v = 0; v >= 0; v = v1) {
|
||||
v1 = atomic_cmpxchg(p, v, v + 1);
|
||||
if (likely(v1 == v))
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
int c = atomic_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c < 0))
|
||||
return false;
|
||||
} while (!atomic_try_cmpxchg(v, &c, c + 1));
|
||||
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifndef atomic_dec_unless_positive
|
||||
static inline int atomic_dec_unless_positive(atomic_t *p)
|
||||
static inline bool atomic_dec_unless_positive(atomic_t *v)
|
||||
{
|
||||
int v, v1;
|
||||
for (v = 0; v <= 0; v = v1) {
|
||||
v1 = atomic_cmpxchg(p, v, v - 1);
|
||||
if (likely(v1 == v))
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
int c = atomic_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c > 0))
|
||||
return false;
|
||||
} while (!atomic_try_cmpxchg(v, &c, c - 1));
|
||||
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
|
||||
@ -639,17 +708,14 @@ static inline int atomic_dec_unless_positive(atomic_t *p)
|
||||
#ifndef atomic_dec_if_positive
|
||||
static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
{
|
||||
int c, old, dec;
|
||||
c = atomic_read(v);
|
||||
for (;;) {
|
||||
int dec, c = atomic_read(v);
|
||||
|
||||
do {
|
||||
dec = c - 1;
|
||||
if (unlikely(dec < 0))
|
||||
break;
|
||||
old = atomic_cmpxchg((v), c, dec);
|
||||
if (likely(old == c))
|
||||
break;
|
||||
c = old;
|
||||
}
|
||||
} while (!atomic_try_cmpxchg(v, &c, dec));
|
||||
|
||||
return dec;
|
||||
}
|
||||
#endif
|
||||
@ -693,11 +759,23 @@ static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
#endif
|
||||
#endif /* atomic64_add_return_relaxed */
|
||||
|
||||
#ifndef atomic64_inc
|
||||
#define atomic64_inc(v) atomic64_add(1, (v))
|
||||
#endif
|
||||
|
||||
/* atomic64_inc_return_relaxed */
|
||||
#ifndef atomic64_inc_return_relaxed
|
||||
|
||||
#ifndef atomic64_inc_return
|
||||
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
|
||||
#define atomic64_inc_return_relaxed(v) atomic64_add_return_relaxed(1, (v))
|
||||
#define atomic64_inc_return_acquire(v) atomic64_add_return_acquire(1, (v))
|
||||
#define atomic64_inc_return_release(v) atomic64_add_return_release(1, (v))
|
||||
#else /* atomic64_inc_return */
|
||||
#define atomic64_inc_return_relaxed atomic64_inc_return
|
||||
#define atomic64_inc_return_acquire atomic64_inc_return
|
||||
#define atomic64_inc_return_release atomic64_inc_return
|
||||
#endif /* atomic64_inc_return */
|
||||
|
||||
#else /* atomic64_inc_return_relaxed */
|
||||
|
||||
@ -742,11 +820,23 @@ static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
#endif
|
||||
#endif /* atomic64_sub_return_relaxed */
|
||||
|
||||
#ifndef atomic64_dec
|
||||
#define atomic64_dec(v) atomic64_sub(1, (v))
|
||||
#endif
|
||||
|
||||
/* atomic64_dec_return_relaxed */
|
||||
#ifndef atomic64_dec_return_relaxed
|
||||
|
||||
#ifndef atomic64_dec_return
|
||||
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
|
||||
#define atomic64_dec_return_relaxed(v) atomic64_sub_return_relaxed(1, (v))
|
||||
#define atomic64_dec_return_acquire(v) atomic64_sub_return_acquire(1, (v))
|
||||
#define atomic64_dec_return_release(v) atomic64_sub_return_release(1, (v))
|
||||
#else /* atomic64_dec_return */
|
||||
#define atomic64_dec_return_relaxed atomic64_dec_return
|
||||
#define atomic64_dec_return_acquire atomic64_dec_return
|
||||
#define atomic64_dec_return_release atomic64_dec_return
|
||||
#endif /* atomic64_dec_return */
|
||||
|
||||
#else /* atomic64_dec_return_relaxed */
|
||||
|
||||
@ -927,12 +1017,22 @@ static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
#endif
|
||||
#endif /* atomic64_fetch_and_relaxed */
|
||||
|
||||
#ifdef atomic64_andnot
|
||||
/* atomic64_fetch_andnot_relaxed */
|
||||
#ifndef atomic64_andnot
|
||||
#define atomic64_andnot(i, v) atomic64_and(~(long long)(i), (v))
|
||||
#endif
|
||||
|
||||
#ifndef atomic64_fetch_andnot_relaxed
|
||||
#define atomic64_fetch_andnot_relaxed atomic64_fetch_andnot
|
||||
#define atomic64_fetch_andnot_acquire atomic64_fetch_andnot
|
||||
#define atomic64_fetch_andnot_release atomic64_fetch_andnot
|
||||
|
||||
#ifndef atomic64_fetch_andnot
|
||||
#define atomic64_fetch_andnot(i, v) atomic64_fetch_and(~(long long)(i), (v))
|
||||
#define atomic64_fetch_andnot_relaxed(i, v) atomic64_fetch_and_relaxed(~(long long)(i), (v))
|
||||
#define atomic64_fetch_andnot_acquire(i, v) atomic64_fetch_and_acquire(~(long long)(i), (v))
|
||||
#define atomic64_fetch_andnot_release(i, v) atomic64_fetch_and_release(~(long long)(i), (v))
|
||||
#else /* atomic64_fetch_andnot */
|
||||
#define atomic64_fetch_andnot_relaxed atomic64_fetch_andnot
|
||||
#define atomic64_fetch_andnot_acquire atomic64_fetch_andnot
|
||||
#define atomic64_fetch_andnot_release atomic64_fetch_andnot
|
||||
#endif /* atomic64_fetch_andnot */
|
||||
|
||||
#else /* atomic64_fetch_andnot_relaxed */
|
||||
|
||||
@ -951,7 +1051,6 @@ static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
__atomic_op_fence(atomic64_fetch_andnot, __VA_ARGS__)
|
||||
#endif
|
||||
#endif /* atomic64_fetch_andnot_relaxed */
|
||||
#endif /* atomic64_andnot */
|
||||
|
||||
/* atomic64_fetch_xor_relaxed */
|
||||
#ifndef atomic64_fetch_xor_relaxed
|
||||
@ -1049,30 +1148,164 @@ static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
#define atomic64_try_cmpxchg_release atomic64_try_cmpxchg
|
||||
#endif /* atomic64_try_cmpxchg */
|
||||
|
||||
#ifndef atomic64_andnot
|
||||
static inline void atomic64_andnot(long long i, atomic64_t *v)
|
||||
/**
|
||||
* atomic64_fetch_add_unless - add unless the number is already a given value
|
||||
* @v: pointer of type atomic64_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, if @v was not already @u.
|
||||
* Returns the original value of @v.
|
||||
*/
|
||||
#ifndef atomic64_fetch_add_unless
|
||||
static inline long long atomic64_fetch_add_unless(atomic64_t *v, long long a,
|
||||
long long u)
|
||||
{
|
||||
atomic64_and(~i, v);
|
||||
long long c = atomic64_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c == u))
|
||||
break;
|
||||
} while (!atomic64_try_cmpxchg(v, &c, c + a));
|
||||
|
||||
return c;
|
||||
}
|
||||
#endif
|
||||
|
||||
/**
|
||||
* atomic64_add_unless - add unless the number is already a given value
|
||||
* @v: pointer of type atomic_t
|
||||
* @a: the amount to add to v...
|
||||
* @u: ...unless v is equal to u.
|
||||
*
|
||||
* Atomically adds @a to @v, if @v was not already @u.
|
||||
* Returns true if the addition was done.
|
||||
*/
|
||||
static inline bool atomic64_add_unless(atomic64_t *v, long long a, long long u)
|
||||
{
|
||||
return atomic64_fetch_add_unless(v, a, u) != u;
|
||||
}
|
||||
|
||||
static inline long long atomic64_fetch_andnot(long long i, atomic64_t *v)
|
||||
{
|
||||
return atomic64_fetch_and(~i, v);
|
||||
}
|
||||
/**
|
||||
* atomic64_inc_not_zero - increment unless the number is zero
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1, if @v is non-zero.
|
||||
* Returns true if the increment was done.
|
||||
*/
|
||||
#ifndef atomic64_inc_not_zero
|
||||
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
|
||||
#endif
|
||||
|
||||
static inline long long atomic64_fetch_andnot_relaxed(long long i, atomic64_t *v)
|
||||
/**
|
||||
* atomic64_inc_and_test - increment and test
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically increments @v by 1
|
||||
* and returns true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#ifndef atomic64_inc_and_test
|
||||
static inline bool atomic64_inc_and_test(atomic64_t *v)
|
||||
{
|
||||
return atomic64_fetch_and_relaxed(~i, v);
|
||||
return atomic64_inc_return(v) == 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
static inline long long atomic64_fetch_andnot_acquire(long long i, atomic64_t *v)
|
||||
/**
|
||||
* atomic64_dec_and_test - decrement and test
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically decrements @v by 1 and
|
||||
* returns true if the result is 0, or false for all other
|
||||
* cases.
|
||||
*/
|
||||
#ifndef atomic64_dec_and_test
|
||||
static inline bool atomic64_dec_and_test(atomic64_t *v)
|
||||
{
|
||||
return atomic64_fetch_and_acquire(~i, v);
|
||||
return atomic64_dec_return(v) == 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v)
|
||||
/**
|
||||
* atomic64_sub_and_test - subtract value from variable and test result
|
||||
* @i: integer value to subtract
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically subtracts @i from @v and returns
|
||||
* true if the result is zero, or false for all
|
||||
* other cases.
|
||||
*/
|
||||
#ifndef atomic64_sub_and_test
|
||||
static inline bool atomic64_sub_and_test(long long i, atomic64_t *v)
|
||||
{
|
||||
return atomic64_fetch_and_release(~i, v);
|
||||
return atomic64_sub_return(i, v) == 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
/**
|
||||
* atomic64_add_negative - add and test if negative
|
||||
* @i: integer value to add
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* Atomically adds @i to @v and returns true
|
||||
* if the result is negative, or false when
|
||||
* result is greater than or equal to zero.
|
||||
*/
|
||||
#ifndef atomic64_add_negative
|
||||
static inline bool atomic64_add_negative(long long i, atomic64_t *v)
|
||||
{
|
||||
return atomic64_add_return(i, v) < 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifndef atomic64_inc_unless_negative
|
||||
static inline bool atomic64_inc_unless_negative(atomic64_t *v)
|
||||
{
|
||||
long long c = atomic64_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c < 0))
|
||||
return false;
|
||||
} while (!atomic64_try_cmpxchg(v, &c, c + 1));
|
||||
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifndef atomic64_dec_unless_positive
|
||||
static inline bool atomic64_dec_unless_positive(atomic64_t *v)
|
||||
{
|
||||
long long c = atomic64_read(v);
|
||||
|
||||
do {
|
||||
if (unlikely(c > 0))
|
||||
return false;
|
||||
} while (!atomic64_try_cmpxchg(v, &c, c - 1));
|
||||
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
|
||||
/*
|
||||
* atomic64_dec_if_positive - decrement by 1 if old value positive
|
||||
* @v: pointer of type atomic64_t
|
||||
*
|
||||
* The function returns the old value of *v minus 1, even if
|
||||
* the atomic64 variable, v, was not decremented.
|
||||
*/
|
||||
#ifndef atomic64_dec_if_positive
|
||||
static inline long long atomic64_dec_if_positive(atomic64_t *v)
|
||||
{
|
||||
long long dec, c = atomic64_read(v);
|
||||
|
||||
do {
|
||||
dec = c - 1;
|
||||
if (unlikely(dec < 0))
|
||||
break;
|
||||
} while (!atomic64_try_cmpxchg(v, &c, dec));
|
||||
|
||||
return dec;
|
||||
}
|
||||
#endif
|
||||
|
||||
|
@ -2,29 +2,9 @@
|
||||
#ifndef _LINUX_BITOPS_H
|
||||
#define _LINUX_BITOPS_H
|
||||
#include <asm/types.h>
|
||||
#include <linux/bits.h>
|
||||
|
||||
#ifdef __KERNEL__
|
||||
#define BIT(nr) (1UL << (nr))
|
||||
#define BIT_ULL(nr) (1ULL << (nr))
|
||||
#define BIT_MASK(nr) (1UL << ((nr) % BITS_PER_LONG))
|
||||
#define BIT_WORD(nr) ((nr) / BITS_PER_LONG)
|
||||
#define BIT_ULL_MASK(nr) (1ULL << ((nr) % BITS_PER_LONG_LONG))
|
||||
#define BIT_ULL_WORD(nr) ((nr) / BITS_PER_LONG_LONG)
|
||||
#define BITS_PER_BYTE 8
|
||||
#define BITS_TO_LONGS(nr) DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(long))
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Create a contiguous bitmask starting at bit position @l and ending at
|
||||
* position @h. For example
|
||||
* GENMASK_ULL(39, 21) gives us the 64bit vector 0x000000ffffe00000.
|
||||
*/
|
||||
#define GENMASK(h, l) \
|
||||
(((~0UL) - (1UL << (l)) + 1) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
|
||||
|
||||
#define GENMASK_ULL(h, l) \
|
||||
(((~0ULL) - (1ULL << (l)) + 1) & \
|
||||
(~0ULL >> (BITS_PER_LONG_LONG - 1 - (h))))
|
||||
|
||||
extern unsigned int __sw_hweight8(unsigned int w);
|
||||
extern unsigned int __sw_hweight16(unsigned int w);
|
||||
|
26
include/linux/bits.h
Normal file
26
include/linux/bits.h
Normal file
@ -0,0 +1,26 @@
|
||||
/* SPDX-License-Identifier: GPL-2.0 */
|
||||
#ifndef __LINUX_BITS_H
|
||||
#define __LINUX_BITS_H
|
||||
#include <asm/bitsperlong.h>
|
||||
|
||||
#define BIT(nr) (1UL << (nr))
|
||||
#define BIT_ULL(nr) (1ULL << (nr))
|
||||
#define BIT_MASK(nr) (1UL << ((nr) % BITS_PER_LONG))
|
||||
#define BIT_WORD(nr) ((nr) / BITS_PER_LONG)
|
||||
#define BIT_ULL_MASK(nr) (1ULL << ((nr) % BITS_PER_LONG_LONG))
|
||||
#define BIT_ULL_WORD(nr) ((nr) / BITS_PER_LONG_LONG)
|
||||
#define BITS_PER_BYTE 8
|
||||
|
||||
/*
|
||||
* Create a contiguous bitmask starting at bit position @l and ending at
|
||||
* position @h. For example
|
||||
* GENMASK_ULL(39, 21) gives us the 64bit vector 0x000000ffffe00000.
|
||||
*/
|
||||
#define GENMASK(h, l) \
|
||||
(((~0UL) - (1UL << (l)) + 1) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
|
||||
|
||||
#define GENMASK_ULL(h, l) \
|
||||
(((~0ULL) - (1ULL << (l)) + 1) & \
|
||||
(~0ULL >> (BITS_PER_LONG_LONG - 1 - (h))))
|
||||
|
||||
#endif /* __LINUX_BITS_H */
|
@ -3,9 +3,10 @@
|
||||
#define _LINUX_REFCOUNT_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/mutex.h>
|
||||
#include <linux/spinlock.h>
|
||||
#include <linux/kernel.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/spinlock_types.h>
|
||||
|
||||
struct mutex;
|
||||
|
||||
/**
|
||||
* struct refcount_t - variant of atomic_t specialized for reference counts
|
||||
@ -42,17 +43,30 @@ static inline unsigned int refcount_read(const refcount_t *r)
|
||||
return atomic_read(&r->refs);
|
||||
}
|
||||
|
||||
extern __must_check bool refcount_add_not_zero_checked(unsigned int i, refcount_t *r);
|
||||
extern void refcount_add_checked(unsigned int i, refcount_t *r);
|
||||
|
||||
extern __must_check bool refcount_inc_not_zero_checked(refcount_t *r);
|
||||
extern void refcount_inc_checked(refcount_t *r);
|
||||
|
||||
extern __must_check bool refcount_sub_and_test_checked(unsigned int i, refcount_t *r);
|
||||
|
||||
extern __must_check bool refcount_dec_and_test_checked(refcount_t *r);
|
||||
extern void refcount_dec_checked(refcount_t *r);
|
||||
|
||||
#ifdef CONFIG_REFCOUNT_FULL
|
||||
extern __must_check bool refcount_add_not_zero(unsigned int i, refcount_t *r);
|
||||
extern void refcount_add(unsigned int i, refcount_t *r);
|
||||
|
||||
extern __must_check bool refcount_inc_not_zero(refcount_t *r);
|
||||
extern void refcount_inc(refcount_t *r);
|
||||
#define refcount_add_not_zero refcount_add_not_zero_checked
|
||||
#define refcount_add refcount_add_checked
|
||||
|
||||
extern __must_check bool refcount_sub_and_test(unsigned int i, refcount_t *r);
|
||||
#define refcount_inc_not_zero refcount_inc_not_zero_checked
|
||||
#define refcount_inc refcount_inc_checked
|
||||
|
||||
#define refcount_sub_and_test refcount_sub_and_test_checked
|
||||
|
||||
#define refcount_dec_and_test refcount_dec_and_test_checked
|
||||
#define refcount_dec refcount_dec_checked
|
||||
|
||||
extern __must_check bool refcount_dec_and_test(refcount_t *r);
|
||||
extern void refcount_dec(refcount_t *r);
|
||||
#else
|
||||
# ifdef CONFIG_ARCH_HAS_REFCOUNT
|
||||
# include <asm/refcount.h>
|
||||
|
@ -167,8 +167,8 @@ struct task_group;
|
||||
* need_sleep = false;
|
||||
* wake_up_state(p, TASK_UNINTERRUPTIBLE);
|
||||
*
|
||||
* Where wake_up_state() (and all other wakeup primitives) imply enough
|
||||
* barriers to order the store of the variable against wakeup.
|
||||
* where wake_up_state() executes a full memory barrier before accessing the
|
||||
* task state.
|
||||
*
|
||||
* Wakeup will do: if (@state & p->state) p->state = TASK_RUNNING, that is,
|
||||
* once it observes the TASK_UNINTERRUPTIBLE store the waking CPU can issue a
|
||||
|
@ -114,29 +114,48 @@ do { \
|
||||
#endif /*arch_spin_is_contended*/
|
||||
|
||||
/*
|
||||
* This barrier must provide two things:
|
||||
* smp_mb__after_spinlock() provides the equivalent of a full memory barrier
|
||||
* between program-order earlier lock acquisitions and program-order later
|
||||
* memory accesses.
|
||||
*
|
||||
* - it must guarantee a STORE before the spin_lock() is ordered against a
|
||||
* LOAD after it, see the comments at its two usage sites.
|
||||
* This guarantees that the following two properties hold:
|
||||
*
|
||||
* - it must ensure the critical section is RCsc.
|
||||
* 1) Given the snippet:
|
||||
*
|
||||
* The latter is important for cases where we observe values written by other
|
||||
* CPUs in spin-loops, without barriers, while being subject to scheduling.
|
||||
* { X = 0; Y = 0; }
|
||||
*
|
||||
* CPU0 CPU1 CPU2
|
||||
* CPU0 CPU1
|
||||
*
|
||||
* for (;;) {
|
||||
* if (READ_ONCE(X))
|
||||
* break;
|
||||
* }
|
||||
* X=1
|
||||
* <sched-out>
|
||||
* <sched-in>
|
||||
* r = X;
|
||||
* WRITE_ONCE(X, 1); WRITE_ONCE(Y, 1);
|
||||
* spin_lock(S); smp_mb();
|
||||
* smp_mb__after_spinlock(); r1 = READ_ONCE(X);
|
||||
* r0 = READ_ONCE(Y);
|
||||
* spin_unlock(S);
|
||||
*
|
||||
* without transitivity it could be that CPU1 observes X!=0 breaks the loop,
|
||||
* we get migrated and CPU2 sees X==0.
|
||||
* it is forbidden that CPU0 does not observe CPU1's store to Y (r0 = 0)
|
||||
* and CPU1 does not observe CPU0's store to X (r1 = 0); see the comments
|
||||
* preceding the call to smp_mb__after_spinlock() in __schedule() and in
|
||||
* try_to_wake_up().
|
||||
*
|
||||
* 2) Given the snippet:
|
||||
*
|
||||
* { X = 0; Y = 0; }
|
||||
*
|
||||
* CPU0 CPU1 CPU2
|
||||
*
|
||||
* spin_lock(S); spin_lock(S); r1 = READ_ONCE(Y);
|
||||
* WRITE_ONCE(X, 1); smp_mb__after_spinlock(); smp_rmb();
|
||||
* spin_unlock(S); r0 = READ_ONCE(X); r2 = READ_ONCE(X);
|
||||
* WRITE_ONCE(Y, 1);
|
||||
* spin_unlock(S);
|
||||
*
|
||||
* it is forbidden that CPU0's critical section executes before CPU1's
|
||||
* critical section (r0 = 1), CPU2 observes CPU1's store to Y (r1 = 1)
|
||||
* and CPU2 does not observe CPU0's store to X (r2 = 0); see the comments
|
||||
* preceding the calls to smp_rmb() in try_to_wake_up() for similar
|
||||
* snippets but "projected" onto two CPUs.
|
||||
*
|
||||
* Property (2) upgrades the lock to an RCsc lock.
|
||||
*
|
||||
* Since most load-store architectures implement ACQUIRE with an smp_mb() after
|
||||
* the LL/SC loop, they need no further barriers. Similarly all our TSO
|
||||
|
@ -575,7 +575,7 @@ static struct bpf_map *bpf_map_inc_not_zero(struct bpf_map *map,
|
||||
{
|
||||
int refold;
|
||||
|
||||
refold = __atomic_add_unless(&map->refcnt, 1, 0);
|
||||
refold = atomic_fetch_add_unless(&map->refcnt, 1, 0);
|
||||
|
||||
if (refold >= BPF_MAX_REFCNT) {
|
||||
__bpf_map_put(map, false);
|
||||
@ -1144,7 +1144,7 @@ struct bpf_prog *bpf_prog_inc_not_zero(struct bpf_prog *prog)
|
||||
{
|
||||
int refold;
|
||||
|
||||
refold = __atomic_add_unless(&prog->aux->refcnt, 1, 0);
|
||||
refold = atomic_fetch_add_unless(&prog->aux->refcnt, 1, 0);
|
||||
|
||||
if (refold >= BPF_MAX_REFCNT) {
|
||||
__bpf_prog_put(prog, false);
|
||||
|
@ -22,8 +22,8 @@
|
||||
*
|
||||
* See also complete_all(), wait_for_completion() and related routines.
|
||||
*
|
||||
* It may be assumed that this function implies a write memory barrier before
|
||||
* changing the task state if and only if any tasks are woken up.
|
||||
* If this function wakes up a task, it executes a full memory barrier before
|
||||
* accessing the task state.
|
||||
*/
|
||||
void complete(struct completion *x)
|
||||
{
|
||||
@ -44,8 +44,8 @@ EXPORT_SYMBOL(complete);
|
||||
*
|
||||
* This will wake up all threads waiting on this particular completion event.
|
||||
*
|
||||
* It may be assumed that this function implies a write memory barrier before
|
||||
* changing the task state if and only if any tasks are woken up.
|
||||
* If this function wakes up a task, it executes a full memory barrier before
|
||||
* accessing the task state.
|
||||
*
|
||||
* Since complete_all() sets the completion of @x permanently to done
|
||||
* to allow multiple waiters to finish, a call to reinit_completion()
|
||||
|
@ -406,8 +406,8 @@ void wake_q_add(struct wake_q_head *head, struct task_struct *task)
|
||||
* its already queued (either by us or someone else) and will get the
|
||||
* wakeup due to that.
|
||||
*
|
||||
* This cmpxchg() implies a full barrier, which pairs with the write
|
||||
* barrier implied by the wakeup in wake_up_q().
|
||||
* This cmpxchg() executes a full barrier, which pairs with the full
|
||||
* barrier executed by the wakeup in wake_up_q().
|
||||
*/
|
||||
if (cmpxchg(&node->next, NULL, WAKE_Q_TAIL))
|
||||
return;
|
||||
@ -435,8 +435,8 @@ void wake_up_q(struct wake_q_head *head)
|
||||
task->wake_q.next = NULL;
|
||||
|
||||
/*
|
||||
* wake_up_process() implies a wmb() to pair with the queueing
|
||||
* in wake_q_add() so as not to miss wakeups.
|
||||
* wake_up_process() executes a full barrier, which pairs with
|
||||
* the queueing in wake_q_add() so as not to miss wakeups.
|
||||
*/
|
||||
wake_up_process(task);
|
||||
put_task_struct(task);
|
||||
@ -1859,8 +1859,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
|
||||
* rq(c1)->lock (if not at the same time, then in that order).
|
||||
* C) LOCK of the rq(c1)->lock scheduling in task
|
||||
*
|
||||
* Transitivity guarantees that B happens after A and C after B.
|
||||
* Note: we only require RCpc transitivity.
|
||||
* Release/acquire chaining guarantees that B happens after A and C after B.
|
||||
* Note: the CPU doing B need not be c0 or c1
|
||||
*
|
||||
* Example:
|
||||
@ -1922,16 +1921,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
|
||||
* UNLOCK rq(0)->lock
|
||||
*
|
||||
*
|
||||
* However; for wakeups there is a second guarantee we must provide, namely we
|
||||
* must observe the state that lead to our wakeup. That is, not only must our
|
||||
* task observe its own prior state, it must also observe the stores prior to
|
||||
* its wakeup.
|
||||
*
|
||||
* This means that any means of doing remote wakeups must order the CPU doing
|
||||
* the wakeup against the CPU the task is going to end up running on. This,
|
||||
* however, is already required for the regular Program-Order guarantee above,
|
||||
* since the waking CPU is the one issueing the ACQUIRE (smp_cond_load_acquire).
|
||||
*
|
||||
* However, for wakeups there is a second guarantee we must provide, namely we
|
||||
* must ensure that CONDITION=1 done by the caller can not be reordered with
|
||||
* accesses to the task state; see try_to_wake_up() and set_current_state().
|
||||
*/
|
||||
|
||||
/**
|
||||
@ -1947,6 +1939,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
|
||||
* Atomic against schedule() which would dequeue a task, also see
|
||||
* set_current_state().
|
||||
*
|
||||
* This function executes a full memory barrier before accessing the task
|
||||
* state; see set_current_state().
|
||||
*
|
||||
* Return: %true if @p->state changes (an actual wakeup was done),
|
||||
* %false otherwise.
|
||||
*/
|
||||
@ -1978,21 +1973,20 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
|
||||
* be possible to, falsely, observe p->on_rq == 0 and get stuck
|
||||
* in smp_cond_load_acquire() below.
|
||||
*
|
||||
* sched_ttwu_pending() try_to_wake_up()
|
||||
* [S] p->on_rq = 1; [L] P->state
|
||||
* UNLOCK rq->lock -----.
|
||||
* \
|
||||
* +--- RMB
|
||||
* schedule() /
|
||||
* LOCK rq->lock -----'
|
||||
* UNLOCK rq->lock
|
||||
* sched_ttwu_pending() try_to_wake_up()
|
||||
* STORE p->on_rq = 1 LOAD p->state
|
||||
* UNLOCK rq->lock
|
||||
*
|
||||
* __schedule() (switch to task 'p')
|
||||
* LOCK rq->lock smp_rmb();
|
||||
* smp_mb__after_spinlock();
|
||||
* UNLOCK rq->lock
|
||||
*
|
||||
* [task p]
|
||||
* [S] p->state = UNINTERRUPTIBLE [L] p->on_rq
|
||||
* STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq
|
||||
*
|
||||
* Pairs with the UNLOCK+LOCK on rq->lock from the
|
||||
* last wakeup of our task and the schedule that got our task
|
||||
* current.
|
||||
* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in
|
||||
* __schedule(). See the comment for smp_mb__after_spinlock().
|
||||
*/
|
||||
smp_rmb();
|
||||
if (p->on_rq && ttwu_remote(p, wake_flags))
|
||||
@ -2006,15 +2000,17 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
|
||||
* One must be running (->on_cpu == 1) in order to remove oneself
|
||||
* from the runqueue.
|
||||
*
|
||||
* [S] ->on_cpu = 1; [L] ->on_rq
|
||||
* UNLOCK rq->lock
|
||||
* RMB
|
||||
* LOCK rq->lock
|
||||
* [S] ->on_rq = 0; [L] ->on_cpu
|
||||
* __schedule() (switch to task 'p') try_to_wake_up()
|
||||
* STORE p->on_cpu = 1 LOAD p->on_rq
|
||||
* UNLOCK rq->lock
|
||||
*
|
||||
* Pairs with the full barrier implied in the UNLOCK+LOCK on rq->lock
|
||||
* from the consecutive calls to schedule(); the first switching to our
|
||||
* task, the second putting it to sleep.
|
||||
* __schedule() (put 'p' to sleep)
|
||||
* LOCK rq->lock smp_rmb();
|
||||
* smp_mb__after_spinlock();
|
||||
* STORE p->on_rq = 0 LOAD p->on_cpu
|
||||
*
|
||||
* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in
|
||||
* __schedule(). See the comment for smp_mb__after_spinlock().
|
||||
*/
|
||||
smp_rmb();
|
||||
|
||||
@ -2120,8 +2116,7 @@ out:
|
||||
*
|
||||
* Return: 1 if the process was woken up, 0 if it was already running.
|
||||
*
|
||||
* It may be assumed that this function implies a write memory barrier before
|
||||
* changing the task state if and only if any tasks are woken up.
|
||||
* This function executes a full memory barrier before accessing the task state.
|
||||
*/
|
||||
int wake_up_process(struct task_struct *p)
|
||||
{
|
||||
|
@ -134,8 +134,8 @@ static void __wake_up_common_lock(struct wait_queue_head *wq_head, unsigned int
|
||||
* @nr_exclusive: how many wake-one or wake-many threads to wake up
|
||||
* @key: is directly passed to the wakeup function
|
||||
*
|
||||
* It may be assumed that this function implies a write memory barrier before
|
||||
* changing the task state if and only if any tasks are woken up.
|
||||
* If this function wakes up a task, it executes a full memory barrier before
|
||||
* accessing the task state.
|
||||
*/
|
||||
void __wake_up(struct wait_queue_head *wq_head, unsigned int mode,
|
||||
int nr_exclusive, void *key)
|
||||
@ -180,8 +180,8 @@ EXPORT_SYMBOL_GPL(__wake_up_locked_key_bookmark);
|
||||
*
|
||||
* On UP it can prevent extra preemption.
|
||||
*
|
||||
* It may be assumed that this function implies a write memory barrier before
|
||||
* changing the task state if and only if any tasks are woken up.
|
||||
* If this function wakes up a task, it executes a full memory barrier before
|
||||
* accessing the task state.
|
||||
*/
|
||||
void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode,
|
||||
int nr_exclusive, void *key)
|
||||
@ -392,35 +392,36 @@ static inline bool is_kthread_should_stop(void)
|
||||
* if (condition)
|
||||
* break;
|
||||
*
|
||||
* p->state = mode; condition = true;
|
||||
* smp_mb(); // A smp_wmb(); // C
|
||||
* if (!wq_entry->flags & WQ_FLAG_WOKEN) wq_entry->flags |= WQ_FLAG_WOKEN;
|
||||
* schedule() try_to_wake_up();
|
||||
* p->state = TASK_RUNNING; ~~~~~~~~~~~~~~~~~~
|
||||
* wq_entry->flags &= ~WQ_FLAG_WOKEN; condition = true;
|
||||
* smp_mb() // B smp_wmb(); // C
|
||||
* wq_entry->flags |= WQ_FLAG_WOKEN;
|
||||
* }
|
||||
* remove_wait_queue(&wq_head, &wait);
|
||||
* // in wait_woken() // in woken_wake_function()
|
||||
*
|
||||
* p->state = mode; wq_entry->flags |= WQ_FLAG_WOKEN;
|
||||
* smp_mb(); // A try_to_wake_up():
|
||||
* if (!(wq_entry->flags & WQ_FLAG_WOKEN)) <full barrier>
|
||||
* schedule() if (p->state & mode)
|
||||
* p->state = TASK_RUNNING; p->state = TASK_RUNNING;
|
||||
* wq_entry->flags &= ~WQ_FLAG_WOKEN; ~~~~~~~~~~~~~~~~~~
|
||||
* smp_mb(); // B condition = true;
|
||||
* } smp_mb(); // C
|
||||
* remove_wait_queue(&wq_head, &wait); wq_entry->flags |= WQ_FLAG_WOKEN;
|
||||
*/
|
||||
long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout)
|
||||
{
|
||||
set_current_state(mode); /* A */
|
||||
/*
|
||||
* The above implies an smp_mb(), which matches with the smp_wmb() from
|
||||
* woken_wake_function() such that if we observe WQ_FLAG_WOKEN we must
|
||||
* also observe all state before the wakeup.
|
||||
* The below executes an smp_mb(), which matches with the full barrier
|
||||
* executed by the try_to_wake_up() in woken_wake_function() such that
|
||||
* either we see the store to wq_entry->flags in woken_wake_function()
|
||||
* or woken_wake_function() sees our store to current->state.
|
||||
*/
|
||||
set_current_state(mode); /* A */
|
||||
if (!(wq_entry->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop())
|
||||
timeout = schedule_timeout(timeout);
|
||||
__set_current_state(TASK_RUNNING);
|
||||
|
||||
/*
|
||||
* The below implies an smp_mb(), it too pairs with the smp_wmb() from
|
||||
* woken_wake_function() such that we must either observe the wait
|
||||
* condition being true _OR_ WQ_FLAG_WOKEN such that we will not miss
|
||||
* an event.
|
||||
* The below executes an smp_mb(), which matches with the smp_mb() (C)
|
||||
* in woken_wake_function() such that either we see the wait condition
|
||||
* being true or the store to wq_entry->flags in woken_wake_function()
|
||||
* follows ours in the coherence order.
|
||||
*/
|
||||
smp_store_mb(wq_entry->flags, wq_entry->flags & ~WQ_FLAG_WOKEN); /* B */
|
||||
|
||||
@ -430,14 +431,8 @@ EXPORT_SYMBOL(wait_woken);
|
||||
|
||||
int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key)
|
||||
{
|
||||
/*
|
||||
* Although this function is called under waitqueue lock, LOCK
|
||||
* doesn't imply write barrier and the users expects write
|
||||
* barrier semantics on wakeup functions. The following
|
||||
* smp_wmb() is equivalent to smp_wmb() in try_to_wake_up()
|
||||
* and is paired with smp_store_mb() in wait_woken().
|
||||
*/
|
||||
smp_wmb(); /* C */
|
||||
/* Pairs with the smp_store_mb() in wait_woken(). */
|
||||
smp_mb(); /* C */
|
||||
wq_entry->flags |= WQ_FLAG_WOKEN;
|
||||
|
||||
return default_wake_function(wq_entry, mode, sync, key);
|
||||
|
@ -178,18 +178,18 @@ long long atomic64_xchg(atomic64_t *v, long long new)
|
||||
}
|
||||
EXPORT_SYMBOL(atomic64_xchg);
|
||||
|
||||
int atomic64_add_unless(atomic64_t *v, long long a, long long u)
|
||||
long long atomic64_fetch_add_unless(atomic64_t *v, long long a, long long u)
|
||||
{
|
||||
unsigned long flags;
|
||||
raw_spinlock_t *lock = lock_addr(v);
|
||||
int ret = 0;
|
||||
long long val;
|
||||
|
||||
raw_spin_lock_irqsave(lock, flags);
|
||||
if (v->counter != u) {
|
||||
val = v->counter;
|
||||
if (val != u)
|
||||
v->counter += a;
|
||||
ret = 1;
|
||||
}
|
||||
raw_spin_unlock_irqrestore(lock, flags);
|
||||
return ret;
|
||||
|
||||
return val;
|
||||
}
|
||||
EXPORT_SYMBOL(atomic64_add_unless);
|
||||
EXPORT_SYMBOL(atomic64_fetch_add_unless);
|
||||
|
@ -35,13 +35,13 @@
|
||||
*
|
||||
*/
|
||||
|
||||
#include <linux/mutex.h>
|
||||
#include <linux/refcount.h>
|
||||
#include <linux/spinlock.h>
|
||||
#include <linux/bug.h>
|
||||
|
||||
#ifdef CONFIG_REFCOUNT_FULL
|
||||
|
||||
/**
|
||||
* refcount_add_not_zero - add a value to a refcount unless it is 0
|
||||
* refcount_add_not_zero_checked - add a value to a refcount unless it is 0
|
||||
* @i: the value to add to the refcount
|
||||
* @r: the refcount
|
||||
*
|
||||
@ -58,7 +58,7 @@
|
||||
*
|
||||
* Return: false if the passed refcount is 0, true otherwise
|
||||
*/
|
||||
bool refcount_add_not_zero(unsigned int i, refcount_t *r)
|
||||
bool refcount_add_not_zero_checked(unsigned int i, refcount_t *r)
|
||||
{
|
||||
unsigned int new, val = atomic_read(&r->refs);
|
||||
|
||||
@ -79,10 +79,10 @@ bool refcount_add_not_zero(unsigned int i, refcount_t *r)
|
||||
|
||||
return true;
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_add_not_zero);
|
||||
EXPORT_SYMBOL(refcount_add_not_zero_checked);
|
||||
|
||||
/**
|
||||
* refcount_add - add a value to a refcount
|
||||
* refcount_add_checked - add a value to a refcount
|
||||
* @i: the value to add to the refcount
|
||||
* @r: the refcount
|
||||
*
|
||||
@ -97,14 +97,14 @@ EXPORT_SYMBOL(refcount_add_not_zero);
|
||||
* cases, refcount_inc(), or one of its variants, should instead be used to
|
||||
* increment a reference count.
|
||||
*/
|
||||
void refcount_add(unsigned int i, refcount_t *r)
|
||||
void refcount_add_checked(unsigned int i, refcount_t *r)
|
||||
{
|
||||
WARN_ONCE(!refcount_add_not_zero(i, r), "refcount_t: addition on 0; use-after-free.\n");
|
||||
WARN_ONCE(!refcount_add_not_zero_checked(i, r), "refcount_t: addition on 0; use-after-free.\n");
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_add);
|
||||
EXPORT_SYMBOL(refcount_add_checked);
|
||||
|
||||
/**
|
||||
* refcount_inc_not_zero - increment a refcount unless it is 0
|
||||
* refcount_inc_not_zero_checked - increment a refcount unless it is 0
|
||||
* @r: the refcount to increment
|
||||
*
|
||||
* Similar to atomic_inc_not_zero(), but will saturate at UINT_MAX and WARN.
|
||||
@ -115,7 +115,7 @@ EXPORT_SYMBOL(refcount_add);
|
||||
*
|
||||
* Return: true if the increment was successful, false otherwise
|
||||
*/
|
||||
bool refcount_inc_not_zero(refcount_t *r)
|
||||
bool refcount_inc_not_zero_checked(refcount_t *r)
|
||||
{
|
||||
unsigned int new, val = atomic_read(&r->refs);
|
||||
|
||||
@ -134,10 +134,10 @@ bool refcount_inc_not_zero(refcount_t *r)
|
||||
|
||||
return true;
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_inc_not_zero);
|
||||
EXPORT_SYMBOL(refcount_inc_not_zero_checked);
|
||||
|
||||
/**
|
||||
* refcount_inc - increment a refcount
|
||||
* refcount_inc_checked - increment a refcount
|
||||
* @r: the refcount to increment
|
||||
*
|
||||
* Similar to atomic_inc(), but will saturate at UINT_MAX and WARN.
|
||||
@ -148,14 +148,14 @@ EXPORT_SYMBOL(refcount_inc_not_zero);
|
||||
* Will WARN if the refcount is 0, as this represents a possible use-after-free
|
||||
* condition.
|
||||
*/
|
||||
void refcount_inc(refcount_t *r)
|
||||
void refcount_inc_checked(refcount_t *r)
|
||||
{
|
||||
WARN_ONCE(!refcount_inc_not_zero(r), "refcount_t: increment on 0; use-after-free.\n");
|
||||
WARN_ONCE(!refcount_inc_not_zero_checked(r), "refcount_t: increment on 0; use-after-free.\n");
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_inc);
|
||||
EXPORT_SYMBOL(refcount_inc_checked);
|
||||
|
||||
/**
|
||||
* refcount_sub_and_test - subtract from a refcount and test if it is 0
|
||||
* refcount_sub_and_test_checked - subtract from a refcount and test if it is 0
|
||||
* @i: amount to subtract from the refcount
|
||||
* @r: the refcount
|
||||
*
|
||||
@ -174,7 +174,7 @@ EXPORT_SYMBOL(refcount_inc);
|
||||
*
|
||||
* Return: true if the resulting refcount is 0, false otherwise
|
||||
*/
|
||||
bool refcount_sub_and_test(unsigned int i, refcount_t *r)
|
||||
bool refcount_sub_and_test_checked(unsigned int i, refcount_t *r)
|
||||
{
|
||||
unsigned int new, val = atomic_read(&r->refs);
|
||||
|
||||
@ -192,10 +192,10 @@ bool refcount_sub_and_test(unsigned int i, refcount_t *r)
|
||||
|
||||
return !new;
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_sub_and_test);
|
||||
EXPORT_SYMBOL(refcount_sub_and_test_checked);
|
||||
|
||||
/**
|
||||
* refcount_dec_and_test - decrement a refcount and test if it is 0
|
||||
* refcount_dec_and_test_checked - decrement a refcount and test if it is 0
|
||||
* @r: the refcount
|
||||
*
|
||||
* Similar to atomic_dec_and_test(), it will WARN on underflow and fail to
|
||||
@ -207,14 +207,14 @@ EXPORT_SYMBOL(refcount_sub_and_test);
|
||||
*
|
||||
* Return: true if the resulting refcount is 0, false otherwise
|
||||
*/
|
||||
bool refcount_dec_and_test(refcount_t *r)
|
||||
bool refcount_dec_and_test_checked(refcount_t *r)
|
||||
{
|
||||
return refcount_sub_and_test(1, r);
|
||||
return refcount_sub_and_test_checked(1, r);
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_dec_and_test);
|
||||
EXPORT_SYMBOL(refcount_dec_and_test_checked);
|
||||
|
||||
/**
|
||||
* refcount_dec - decrement a refcount
|
||||
* refcount_dec_checked - decrement a refcount
|
||||
* @r: the refcount
|
||||
*
|
||||
* Similar to atomic_dec(), it will WARN on underflow and fail to decrement
|
||||
@ -223,12 +223,11 @@ EXPORT_SYMBOL(refcount_dec_and_test);
|
||||
* Provides release memory ordering, such that prior loads and stores are done
|
||||
* before.
|
||||
*/
|
||||
void refcount_dec(refcount_t *r)
|
||||
void refcount_dec_checked(refcount_t *r)
|
||||
{
|
||||
WARN_ONCE(refcount_dec_and_test(r), "refcount_t: decrement hit 0; leaking memory.\n");
|
||||
WARN_ONCE(refcount_dec_and_test_checked(r), "refcount_t: decrement hit 0; leaking memory.\n");
|
||||
}
|
||||
EXPORT_SYMBOL(refcount_dec);
|
||||
#endif /* CONFIG_REFCOUNT_FULL */
|
||||
EXPORT_SYMBOL(refcount_dec_checked);
|
||||
|
||||
/**
|
||||
* refcount_dec_if_one - decrement a refcount if it is 1
|
||||
|
@ -244,7 +244,7 @@ static int pppoatm_may_send(struct pppoatm_vcc *pvcc, int size)
|
||||
* the packet count limit, so...
|
||||
*/
|
||||
if (atm_may_send(pvcc->atmvcc, size) &&
|
||||
atomic_inc_not_zero_hint(&pvcc->inflight, NONE_INFLIGHT))
|
||||
atomic_inc_not_zero(&pvcc->inflight))
|
||||
return 1;
|
||||
|
||||
/*
|
||||
|
@ -415,7 +415,7 @@ void rxrpc_incoming_call(struct rxrpc_sock *rx,
|
||||
bool rxrpc_queue_call(struct rxrpc_call *call)
|
||||
{
|
||||
const void *here = __builtin_return_address(0);
|
||||
int n = __atomic_add_unless(&call->usage, 1, 0);
|
||||
int n = atomic_fetch_add_unless(&call->usage, 1, 0);
|
||||
if (n == 0)
|
||||
return false;
|
||||
if (rxrpc_queue_work(&call->processor))
|
||||
|
@ -266,7 +266,7 @@ void rxrpc_kill_connection(struct rxrpc_connection *conn)
|
||||
bool rxrpc_queue_conn(struct rxrpc_connection *conn)
|
||||
{
|
||||
const void *here = __builtin_return_address(0);
|
||||
int n = __atomic_add_unless(&conn->usage, 1, 0);
|
||||
int n = atomic_fetch_add_unless(&conn->usage, 1, 0);
|
||||
if (n == 0)
|
||||
return false;
|
||||
if (rxrpc_queue_work(&conn->processor))
|
||||
@ -309,7 +309,7 @@ rxrpc_get_connection_maybe(struct rxrpc_connection *conn)
|
||||
const void *here = __builtin_return_address(0);
|
||||
|
||||
if (conn) {
|
||||
int n = __atomic_add_unless(&conn->usage, 1, 0);
|
||||
int n = atomic_fetch_add_unless(&conn->usage, 1, 0);
|
||||
if (n > 0)
|
||||
trace_rxrpc_conn(conn, rxrpc_conn_got, n + 1, here);
|
||||
else
|
||||
|
@ -305,7 +305,7 @@ struct rxrpc_local *rxrpc_get_local_maybe(struct rxrpc_local *local)
|
||||
const void *here = __builtin_return_address(0);
|
||||
|
||||
if (local) {
|
||||
int n = __atomic_add_unless(&local->usage, 1, 0);
|
||||
int n = atomic_fetch_add_unless(&local->usage, 1, 0);
|
||||
if (n > 0)
|
||||
trace_rxrpc_local(local, rxrpc_local_got, n + 1, here);
|
||||
else
|
||||
|
@ -406,7 +406,7 @@ struct rxrpc_peer *rxrpc_get_peer_maybe(struct rxrpc_peer *peer)
|
||||
const void *here = __builtin_return_address(0);
|
||||
|
||||
if (peer) {
|
||||
int n = __atomic_add_unless(&peer->usage, 1, 0);
|
||||
int n = atomic_fetch_add_unless(&peer->usage, 1, 0);
|
||||
if (n > 0)
|
||||
trace_rxrpc_peer(peer, rxrpc_peer_got, n + 1, here);
|
||||
else
|
||||
|
@ -804,7 +804,7 @@ type of fence:
|
||||
Second, some types of fence affect the way the memory subsystem
|
||||
propagates stores. When a fence instruction is executed on CPU C:
|
||||
|
||||
For each other CPU C', smb_wmb() forces all po-earlier stores
|
||||
For each other CPU C', smp_wmb() forces all po-earlier stores
|
||||
on C to propagate to C' before any po-later stores do.
|
||||
|
||||
For each other CPU C', any store which propagates to C before
|
||||
|
@ -126,7 +126,7 @@ However, it is not necessarily the case that accesses ordered by
|
||||
locking will be seen as ordered by CPUs not holding that lock.
|
||||
Consider this example:
|
||||
|
||||
/* See Z6.0+pooncelock+pooncelock+pombonce.litmus. */
|
||||
/* See Z6.0+pooncerelease+poacquirerelease+fencembonceonce.litmus. */
|
||||
void CPU0(void)
|
||||
{
|
||||
spin_lock(&mylock);
|
||||
@ -292,7 +292,7 @@ and to use smp_load_acquire() instead of smp_rmb(). However, the older
|
||||
smp_wmb() and smp_rmb() APIs are still heavily used, so it is important
|
||||
to understand their use cases. The general approach is shown below:
|
||||
|
||||
/* See MP+wmbonceonce+rmbonceonce.litmus. */
|
||||
/* See MP+fencewmbonceonce+fencermbonceonce.litmus. */
|
||||
void CPU0(void)
|
||||
{
|
||||
WRITE_ONCE(x, 1);
|
||||
@ -322,9 +322,9 @@ the following write-side code fragment:
|
||||
And the xlog_valid_lsn() function in fs/xfs/xfs_log_priv.h contains
|
||||
the corresponding read-side code fragment:
|
||||
|
||||
cur_cycle = ACCESS_ONCE(log->l_curr_cycle);
|
||||
cur_cycle = READ_ONCE(log->l_curr_cycle);
|
||||
smp_rmb();
|
||||
cur_block = ACCESS_ONCE(log->l_curr_block);
|
||||
cur_block = READ_ONCE(log->l_curr_block);
|
||||
|
||||
Alternatively, consider the following comment in function
|
||||
perf_output_put_handle() in kernel/events/ring_buffer.c:
|
||||
@ -360,7 +360,7 @@ can be seen in the LB+poonceonces.litmus litmus test.
|
||||
One way of avoiding the counter-intuitive outcome is through the use of a
|
||||
control dependency paired with a full memory barrier:
|
||||
|
||||
/* See LB+ctrlonceonce+mbonceonce.litmus. */
|
||||
/* See LB+fencembonceonce+ctrlonceonce.litmus. */
|
||||
void CPU0(void)
|
||||
{
|
||||
r0 = READ_ONCE(x);
|
||||
@ -476,7 +476,7 @@ that one CPU first stores to one variable and then loads from a second,
|
||||
while another CPU stores to the second variable and then loads from the
|
||||
first. Preserving order requires nothing less than full barriers:
|
||||
|
||||
/* See SB+mbonceonces.litmus. */
|
||||
/* See SB+fencembonceonces.litmus. */
|
||||
void CPU0(void)
|
||||
{
|
||||
WRITE_ONCE(x, 1);
|
||||
|
@ -35,13 +35,13 @@ BASIC USAGE: HERD7
|
||||
The memory model is used, in conjunction with "herd7", to exhaustively
|
||||
explore the state space of small litmus tests.
|
||||
|
||||
For example, to run SB+mbonceonces.litmus against the memory model:
|
||||
For example, to run SB+fencembonceonces.litmus against the memory model:
|
||||
|
||||
$ herd7 -conf linux-kernel.cfg litmus-tests/SB+mbonceonces.litmus
|
||||
$ herd7 -conf linux-kernel.cfg litmus-tests/SB+fencembonceonces.litmus
|
||||
|
||||
Here is the corresponding output:
|
||||
|
||||
Test SB+mbonceonces Allowed
|
||||
Test SB+fencembonceonces Allowed
|
||||
States 3
|
||||
0:r0=0; 1:r0=1;
|
||||
0:r0=1; 1:r0=0;
|
||||
@ -50,8 +50,8 @@ Here is the corresponding output:
|
||||
Witnesses
|
||||
Positive: 0 Negative: 3
|
||||
Condition exists (0:r0=0 /\ 1:r0=0)
|
||||
Observation SB+mbonceonces Never 0 3
|
||||
Time SB+mbonceonces 0.01
|
||||
Observation SB+fencembonceonces Never 0 3
|
||||
Time SB+fencembonceonces 0.01
|
||||
Hash=d66d99523e2cac6b06e66f4c995ebb48
|
||||
|
||||
The "Positive: 0 Negative: 3" and the "Never 0 3" each indicate that
|
||||
@ -67,16 +67,16 @@ BASIC USAGE: KLITMUS7
|
||||
The "klitmus7" tool converts a litmus test into a Linux kernel module,
|
||||
which may then be loaded and run.
|
||||
|
||||
For example, to run SB+mbonceonces.litmus against hardware:
|
||||
For example, to run SB+fencembonceonces.litmus against hardware:
|
||||
|
||||
$ mkdir mymodules
|
||||
$ klitmus7 -o mymodules litmus-tests/SB+mbonceonces.litmus
|
||||
$ klitmus7 -o mymodules litmus-tests/SB+fencembonceonces.litmus
|
||||
$ cd mymodules ; make
|
||||
$ sudo sh run.sh
|
||||
|
||||
The corresponding output includes:
|
||||
|
||||
Test SB+mbonceonces Allowed
|
||||
Test SB+fencembonceonces Allowed
|
||||
Histogram (3 states)
|
||||
644580 :>0:r0=1; 1:r0=0;
|
||||
644328 :>0:r0=0; 1:r0=1;
|
||||
@ -86,8 +86,8 @@ The corresponding output includes:
|
||||
Positive: 0, Negative: 2000000
|
||||
Condition exists (0:r0=0 /\ 1:r0=0) is NOT validated
|
||||
Hash=d66d99523e2cac6b06e66f4c995ebb48
|
||||
Observation SB+mbonceonces Never 0 2000000
|
||||
Time SB+mbonceonces 0.16
|
||||
Observation SB+fencembonceonces Never 0 2000000
|
||||
Time SB+fencembonceonces 0.16
|
||||
|
||||
The "Positive: 0 Negative: 2000000" and the "Never 0 2000000" indicate
|
||||
that during two million trials, the state specified in this litmus
|
||||
|
@ -13,7 +13,7 @@
|
||||
|
||||
"Linux-kernel memory consistency model"
|
||||
|
||||
enum Accesses = 'once (*READ_ONCE,WRITE_ONCE,ACCESS_ONCE*) ||
|
||||
enum Accesses = 'once (*READ_ONCE,WRITE_ONCE*) ||
|
||||
'release (*smp_store_release*) ||
|
||||
'acquire (*smp_load_acquire*) ||
|
||||
'noreturn (* R of non-return RMW *)
|
||||
|
@ -1,4 +1,4 @@
|
||||
C IRIW+mbonceonces+OnceOnce
|
||||
C IRIW+fencembonceonces+OnceOnce
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -1,4 +1,4 @@
|
||||
C ISA2+pooncelock+pooncelock+pombonce.litmus
|
||||
C ISA2+pooncelock+pooncelock+pombonce
|
||||
|
||||
(*
|
||||
* Result: Sometimes
|
||||
|
@ -1,4 +1,4 @@
|
||||
C LB+ctrlonceonce+mbonceonce
|
||||
C LB+fencembonceonce+ctrlonceonce
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -1,4 +1,4 @@
|
||||
C MP+wmbonceonce+rmbonceonce
|
||||
C MP+fencewmbonceonce+fencermbonceonce
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -1,4 +1,4 @@
|
||||
C R+mbonceonces
|
||||
C R+fencembonceonces
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -18,7 +18,7 @@ CoWW+poonceonce.litmus
|
||||
Test of write-write coherence, that is, whether or not two
|
||||
successive writes to the same variable are ordered.
|
||||
|
||||
IRIW+mbonceonces+OnceOnce.litmus
|
||||
IRIW+fencembonceonces+OnceOnce.litmus
|
||||
Test of independent reads from independent writes with smp_mb()
|
||||
between each pairs of reads. In other words, is smp_mb()
|
||||
sufficient to cause two different reading processes to agree on
|
||||
@ -47,7 +47,7 @@ ISA2+pooncerelease+poacquirerelease+poacquireonce.litmus
|
||||
Can a release-acquire chain order a prior store against
|
||||
a later load?
|
||||
|
||||
LB+ctrlonceonce+mbonceonce.litmus
|
||||
LB+fencembonceonce+ctrlonceonce.litmus
|
||||
Does a control dependency and an smp_mb() suffice for the
|
||||
load-buffering litmus test, where each process reads from one
|
||||
of two variables then writes to the other?
|
||||
@ -88,14 +88,14 @@ MP+porevlocks.litmus
|
||||
As below, but with the first access of the writer process
|
||||
and the second access of reader process protected by a lock.
|
||||
|
||||
MP+wmbonceonce+rmbonceonce.litmus
|
||||
MP+fencewmbonceonce+fencermbonceonce.litmus
|
||||
Does a smp_wmb() (between the stores) and an smp_rmb() (between
|
||||
the loads) suffice for the message-passing litmus test, where one
|
||||
process writes data and then a flag, and the other process reads
|
||||
the flag and then the data. (This is similar to the ISA2 tests,
|
||||
but with two processes instead of three.)
|
||||
|
||||
R+mbonceonces.litmus
|
||||
R+fencembonceonces.litmus
|
||||
This is the fully ordered (via smp_mb()) version of one of
|
||||
the classic counterintuitive litmus tests that illustrates the
|
||||
effects of store propagation delays.
|
||||
@ -103,7 +103,7 @@ R+mbonceonces.litmus
|
||||
R+poonceonces.litmus
|
||||
As above, but without the smp_mb() invocations.
|
||||
|
||||
SB+mbonceonces.litmus
|
||||
SB+fencembonceonces.litmus
|
||||
This is the fully ordered (again, via smp_mb() version of store
|
||||
buffering, which forms the core of Dekker's mutual-exclusion
|
||||
algorithm.
|
||||
@ -111,15 +111,24 @@ SB+mbonceonces.litmus
|
||||
SB+poonceonces.litmus
|
||||
As above, but without the smp_mb() invocations.
|
||||
|
||||
SB+rfionceonce-poonceonces.litmus
|
||||
This litmus test demonstrates that LKMM is not fully multicopy
|
||||
atomic. (Neither is it other multicopy atomic.) This litmus test
|
||||
also demonstrates the "locations" debugging aid, which designates
|
||||
additional registers and locations to be printed out in the dump
|
||||
of final states in the herd7 output. Without the "locations"
|
||||
statement, only those registers and locations mentioned in the
|
||||
"exists" clause will be printed.
|
||||
|
||||
S+poonceonces.litmus
|
||||
As below, but without the smp_wmb() and acquire load.
|
||||
|
||||
S+wmbonceonce+poacquireonce.litmus
|
||||
S+fencewmbonceonce+poacquireonce.litmus
|
||||
Can a smp_wmb(), instead of a release, and an acquire order
|
||||
a prior store against a subsequent store?
|
||||
|
||||
WRC+poonceonces+Once.litmus
|
||||
WRC+pooncerelease+rmbonceonce+Once.litmus
|
||||
WRC+pooncerelease+fencermbonceonce+Once.litmus
|
||||
These two are members of an extension of the MP litmus-test
|
||||
class in which the first write is moved to a separate process.
|
||||
The second is forbidden because smp_store_release() is
|
||||
@ -134,7 +143,7 @@ Z6.0+pooncelock+poonceLock+pombonce.litmus
|
||||
As above, but with smp_mb__after_spinlock() immediately
|
||||
following the spin_lock().
|
||||
|
||||
Z6.0+pooncerelease+poacquirerelease+mbonceonce.litmus
|
||||
Z6.0+pooncerelease+poacquirerelease+fencembonceonce.litmus
|
||||
Is the ordering provided by a release-acquire chain sufficient
|
||||
to make ordering apparent to accesses by a process that does
|
||||
not participate in that release-acquire chain?
|
||||
|
@ -1,4 +1,4 @@
|
||||
C S+wmbonceonce+poacquireonce
|
||||
C S+fencewmbonceonce+poacquireonce
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -1,4 +1,4 @@
|
||||
C SB+mbonceonces
|
||||
C SB+fencembonceonces
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -0,0 +1,32 @@
|
||||
C SB+rfionceonce-poonceonces
|
||||
|
||||
(*
|
||||
* Result: Sometimes
|
||||
*
|
||||
* This litmus test demonstrates that LKMM is not fully multicopy atomic.
|
||||
*)
|
||||
|
||||
{}
|
||||
|
||||
P0(int *x, int *y)
|
||||
{
|
||||
int r1;
|
||||
int r2;
|
||||
|
||||
WRITE_ONCE(*x, 1);
|
||||
r1 = READ_ONCE(*x);
|
||||
r2 = READ_ONCE(*y);
|
||||
}
|
||||
|
||||
P1(int *x, int *y)
|
||||
{
|
||||
int r3;
|
||||
int r4;
|
||||
|
||||
WRITE_ONCE(*y, 1);
|
||||
r3 = READ_ONCE(*y);
|
||||
r4 = READ_ONCE(*x);
|
||||
}
|
||||
|
||||
locations [0:r1; 1:r3; x; y] (* Debug aid: Print things not in "exists". *)
|
||||
exists (0:r2=0 /\ 1:r4=0)
|
@ -1,4 +1,4 @@
|
||||
C WRC+pooncerelease+rmbonceonce+Once
|
||||
C WRC+pooncerelease+fencermbonceonce+Once
|
||||
|
||||
(*
|
||||
* Result: Never
|
@ -1,4 +1,4 @@
|
||||
C Z6.0+pooncerelease+poacquirerelease+mbonceonce
|
||||
C Z6.0+pooncerelease+poacquirerelease+fencembonceonce
|
||||
|
||||
(*
|
||||
* Result: Sometimes
|
2
tools/memory-model/scripts/checkalllitmus.sh
Normal file → Executable file
2
tools/memory-model/scripts/checkalllitmus.sh
Normal file → Executable file
@ -9,7 +9,7 @@
|
||||
# appended.
|
||||
#
|
||||
# Usage:
|
||||
# sh checkalllitmus.sh [ directory ]
|
||||
# checkalllitmus.sh [ directory ]
|
||||
#
|
||||
# The LINUX_HERD_OPTIONS environment variable may be used to specify
|
||||
# arguments to herd, whose default is defined by the checklitmus.sh script.
|
||||
|
2
tools/memory-model/scripts/checklitmus.sh
Normal file → Executable file
2
tools/memory-model/scripts/checklitmus.sh
Normal file → Executable file
@ -8,7 +8,7 @@
|
||||
# with ".out" appended.
|
||||
#
|
||||
# Usage:
|
||||
# sh checklitmus.sh file.litmus
|
||||
# checklitmus.sh file.litmus
|
||||
#
|
||||
# The LINUX_HERD_OPTIONS environment variable may be used to specify
|
||||
# arguments to herd, which default to "-conf linux-kernel.cfg". Thus,
|
||||
|
Loading…
Reference in New Issue
Block a user