project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
c4c2592c83
server-skynet-source-3rd-je.../include/jemalloc/internal/atomic_inlines.h

528 lines
14 KiB
C
Raw Normal View History

Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#ifndef JEMALLOC_INTERNAL_ATOMIC_INLINES_H
#define JEMALLOC_INTERNAL_ATOMIC_INLINES_H
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
Add atomic operations tests and fix latent bugs.
2014-08-07 14:36:19 +08:00
/*
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
* All arithmetic functions return the arithmetic result of the atomic
* operation. Some atomic operation APIs return the value prior to mutation, in
* which case the following functions must redundantly compute the result so
* that it can be returned. These functions are normally inlined, so the extra
* operations can be optimized away if the return values aren't used by the
* callers.
Add atomic operations tests and fix latent bugs.
2014-08-07 14:36:19 +08:00
*
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
* <t> atomic_read_<t>(<t> *p) { return (*p); }
Refactor arenas array (fixes deadlock). Refactor the arenas array, which contains pointers to all extant arenas, such that it starts out as a sparse array of maximum size, and use double-checked atomics-based reads as the basis for fast and simple arena_get(). Additionally, reduce arenas_lock's role such that it only protects against arena initalization races. These changes remove the possibility for arena lookups to trigger locking, which resolves at least one known (fork-related) deadlock. This resolves #315.
2016-02-25 15:58:10 +08:00
* <t> atomic_add_<t>(<t> *p, <t> x) { return (*p += x); }
* <t> atomic_sub_<t>(<t> *p, <t> x) { return (*p -= x); }
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
* bool atomic_cas_<t>(<t> *p, <t> c, <t> s)
* {
* if (*p != c)
* return (true);
* *p = s;
* return (false);
* }
* void atomic_write_<t>(<t> *p, <t> x) { *p = x; }
Add atomic operations tests and fix latent bugs.
2014-08-07 14:36:19 +08:00
*/
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
#ifndef JEMALLOC_ENABLE_INLINE
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
# if (LG_SIZEOF_PTR == 3 || LG_SIZEOF_INT == 3)
uint64_t atomic_add_u64(uint64_t *p, uint64_t x);
uint64_t atomic_sub_u64(uint64_t *p, uint64_t x);
bool atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s);
void atomic_write_u64(uint64_t *p, uint64_t x);
# endif
uint32_t atomic_add_u32(uint32_t *p, uint32_t x);
uint32_t atomic_sub_u32(uint32_t *p, uint32_t x);
bool atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s);
void atomic_write_u32(uint32_t *p, uint32_t x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
void *atomic_add_p(void **p, void *x);
void *atomic_sub_p(void **p, void *x);
bool atomic_cas_p(void **p, void *c, void *s);
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
void atomic_write_p(void **p, const void *x);
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
size_t atomic_add_zu(size_t *p, size_t x);
size_t atomic_sub_zu(size_t *p, size_t x);
bool atomic_cas_zu(size_t *p, size_t c, size_t s);
void atomic_write_zu(size_t *p, size_t x);
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
unsigned atomic_add_u(unsigned *p, unsigned x);
unsigned atomic_sub_u(unsigned *p, unsigned x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
bool atomic_cas_u(unsigned *p, unsigned c, unsigned s);
void atomic_write_u(unsigned *p, unsigned x);
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
#endif
#if (defined(JEMALLOC_ENABLE_INLINE) || defined(JEMALLOC_ATOMIC_C_))
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
/******************************************************************************/
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
/* 64-bit operations. */
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
#if (LG_SIZEOF_PTR == 3 || LG_SIZEOF_INT == 3)
# if (defined(__amd64__) || defined(__x86_64__))
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint64_t t = x;
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
asm volatile (
"lock; xaddq %0, %1;"
: "+r" (t), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (t + x);
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
}
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint64_t t;
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
x = (uint64_t)(-(int64_t)x);
t = x;
asm volatile (
"lock; xaddq %0, %1;"
: "+r" (t), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (t + x);
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint8_t success;
Add support for MSVC Tested with MSVC 8 32 and 64 bits.
2012-04-30 18:38:31 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
asm volatile (
"lock; cmpxchgq %4, %0;"
"sete %1;"
: "=m" (*p), "=a" (success) /* Outputs. */
: "m" (*p), "a" (c), "r" (s) /* Inputs. */
: "memory" /* Clobbers. */
);
return (!(bool)success);
Add support for MSVC Tested with MSVC 8 32 and 64 bits.
2012-04-30 18:38:31 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
asm volatile (
Remove unnecessary xchg* lock prefixes.
2015-02-11 08:05:52 +08:00
"xchgq %1, %0;" /* Lock is implied by xchgq. */
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
: "=m" (*p), "+r" (x) /* Outputs. */
: "m" (*p) /* Inputs. */
: "memory" /* Clobbers. */
);
Add support for MSVC Tested with MSVC 8 32 and 64 bits.
2012-04-30 18:38:31 +08:00
}
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
# elif (defined(JEMALLOC_C11ATOMICS))
Add a C11 atomics-based implementation of atomic.h API.
2014-12-06 09:42:41 +08:00
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u64(uint64_t *p, uint64_t x) {
Add a C11 atomics-based implementation of atomic.h API.
2014-12-06 09:42:41 +08:00
volatile atomic_uint_least64_t *a = (volatile atomic_uint_least64_t *)p;
return (atomic_fetch_add(a, x) + x);
}
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u64(uint64_t *p, uint64_t x) {
Add a C11 atomics-based implementation of atomic.h API.
2014-12-06 09:42:41 +08:00
volatile atomic_uint_least64_t *a = (volatile atomic_uint_least64_t *)p;
return (atomic_fetch_sub(a, x) - x);
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s) {
Fix type errors in C11 versions of atomic_*() functions.
2015-05-28 11:31:51 +08:00
volatile atomic_uint_least64_t *a = (volatile atomic_uint_least64_t *)p;
return (!atomic_compare_exchange_strong(a, &c, s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u64(uint64_t *p, uint64_t x) {
Fix type errors in C11 versions of atomic_*() functions.
2015-05-28 11:31:51 +08:00
volatile atomic_uint_least64_t *a = (volatile atomic_uint_least64_t *)p;
atomic_store(a, x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
}
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
# elif (defined(JEMALLOC_ATOMIC9))
Add atomic operation support for OS X.
2011-03-19 10:10:31 +08:00
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
/*
* atomic_fetchadd_64() doesn't exist, but we only ever use this
* function on LP64 systems, so atomic_fetchadd_long() will do.
*/
assert(sizeof(uint64_t) == sizeof(unsigned long));
return (atomic_fetchadd_long(p, (unsigned long)x) + x);
Add atomic operation support for OS X.
2011-03-19 10:10:31 +08:00
}
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
assert(sizeof(uint64_t) == sizeof(unsigned long));
return (atomic_fetchadd_long(p, (unsigned long)(-(long)x)) - x);
}
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
assert(sizeof(uint64_t) == sizeof(unsigned long));
return (!atomic_cmpset_long(p, (unsigned long)c, (unsigned long)s));
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
assert(sizeof(uint64_t) == sizeof(unsigned long));
atomic_store_rel_long(p, x);
Add atomic operation support for OS X.
2011-03-19 10:10:31 +08:00
}
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
# elif (defined(JEMALLOC_OSATOMIC))
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (OSAtomicAdd64((int64_t)x, (int64_t *)p));
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
}
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (OSAtomicAdd64(-((int64_t)x), (int64_t *)p));
}
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (!OSAtomicCompareAndSwap64(c, s, (int64_t *)p));
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint64_t o;
/*The documented OSAtomic*() API does not expose an atomic exchange. */
do {
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
o = atomic_read_u64(p);
} while (atomic_cas_u64(p, o, x));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
}
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
# elif (defined(_MSC_VER))
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (InterlockedExchangeAdd64(p, x) + x);
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
}
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (InterlockedExchangeAdd64(p, -((int64_t)x)) - x);
}
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint64_t o;
o = InterlockedCompareExchange64(p, s, c);
return (o != c);
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
InterlockedExchange64(p, x);
}
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
# elif (defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8) || \
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
defined(JE_FORCE_SYNC_COMPARE_AND_SWAP_8))
Use __sync_add_and_fetch and __sync_sub_and_fetch when they are available These functions may be available as inlines or as libgcc functions. In the former case, a __GCC_HAVE_SYNC_COMPARE_AND_SWAP_n macro is defined. But we still want to use these functions in the latter case, when we don't have our own implementation.
2012-03-26 23:03:41 +08:00
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u64(uint64_t *p, uint64_t x) {
Use __sync_add_and_fetch and __sync_sub_and_fetch when they are available These functions may be available as inlines or as libgcc functions. In the former case, a __GCC_HAVE_SYNC_COMPARE_AND_SWAP_n macro is defined. But we still want to use these functions in the latter case, when we don't have our own implementation.
2012-03-26 23:03:41 +08:00
return (__sync_add_and_fetch(p, x));
}
JEMALLOC_INLINE uint64_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u64(uint64_t *p, uint64_t x) {
Use __sync_add_and_fetch and __sync_sub_and_fetch when they are available These functions may be available as inlines or as libgcc functions. In the former case, a __GCC_HAVE_SYNC_COMPARE_AND_SWAP_n macro is defined. But we still want to use these functions in the latter case, when we don't have our own implementation.
2012-03-26 23:03:41 +08:00
return (__sync_sub_and_fetch(p, x));
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u64(uint64_t *p, uint64_t c, uint64_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (!__sync_bool_compare_and_swap(p, c, s));
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u64(uint64_t *p, uint64_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
__sync_lock_test_and_set(p, x);
}
Revert "Define 64-bits atomics unconditionally" This reverts commit c2942e2c0e097e7c75a3addd0b9c87758f91692e. This resolves #495.
2016-11-08 02:53:35 +08:00
# else
# error "Missing implementation for 64-bit atomic operations"
# endif
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
#endif
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
/******************************************************************************/
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
/* 32-bit operations. */
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (defined(__i386__) || defined(__amd64__) || defined(__x86_64__))
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint32_t t = x;
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
asm volatile (
"lock; xaddl %0, %1;"
: "+r" (t), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (t + x);
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
}
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint32_t t;
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
x = (uint32_t)(-(int32_t)x);
t = x;
asm volatile (
"lock; xaddl %0, %1;"
: "+r" (t), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (t + x);
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint8_t success;
Add support for MSVC Tested with MSVC 8 32 and 64 bits.
2012-04-30 18:38:31 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
asm volatile (
"lock; cmpxchgl %4, %0;"
"sete %1;"
: "=m" (*p), "=a" (success) /* Outputs. */
: "m" (*p), "a" (c), "r" (s) /* Inputs. */
: "memory"
);
return (!(bool)success);
Add support for MSVC Tested with MSVC 8 32 and 64 bits.
2012-04-30 18:38:31 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
asm volatile (
Remove unnecessary xchg* lock prefixes.
2015-02-11 08:05:52 +08:00
"xchgl %1, %0;" /* Lock is implied by xchgl. */
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
: "=m" (*p), "+r" (x) /* Outputs. */
: "m" (*p) /* Inputs. */
: "memory" /* Clobbers. */
);
Add support for MSVC Tested with MSVC 8 32 and 64 bits.
2012-04-30 18:38:31 +08:00
}
Add a C11 atomics-based implementation of atomic.h API.
2014-12-06 09:42:41 +08:00
# elif (defined(JEMALLOC_C11ATOMICS))
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u32(uint32_t *p, uint32_t x) {
Add a C11 atomics-based implementation of atomic.h API.
2014-12-06 09:42:41 +08:00
volatile atomic_uint_least32_t *a = (volatile atomic_uint_least32_t *)p;
return (atomic_fetch_add(a, x) + x);
}
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u32(uint32_t *p, uint32_t x) {
Add a C11 atomics-based implementation of atomic.h API.
2014-12-06 09:42:41 +08:00
volatile atomic_uint_least32_t *a = (volatile atomic_uint_least32_t *)p;
return (atomic_fetch_sub(a, x) - x);
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s) {
Fix type errors in C11 versions of atomic_*() functions.
2015-05-28 11:31:51 +08:00
volatile atomic_uint_least32_t *a = (volatile atomic_uint_least32_t *)p;
return (!atomic_compare_exchange_strong(a, &c, s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u32(uint32_t *p, uint32_t x) {
Fix type errors in C11 versions of atomic_*() functions.
2015-05-28 11:31:51 +08:00
volatile atomic_uint_least32_t *a = (volatile atomic_uint_least32_t *)p;
atomic_store(a, x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
}
#elif (defined(JEMALLOC_ATOMIC9))
Add atomic operation support for OS X.
2011-03-19 10:10:31 +08:00
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (atomic_fetchadd_32(p, x) + x);
Add atomic operation support for OS X.
2011-03-19 10:10:31 +08:00
}
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (atomic_fetchadd_32(p, (uint32_t)(-(int32_t)x)) - x);
Add atomic operation support for OS X.
2011-03-19 10:10:31 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (!atomic_cmpset_32(p, c, s));
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
atomic_store_rel_32(p, x);
}
#elif (defined(JEMALLOC_OSATOMIC))
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (OSAtomicAdd32((int32_t)x, (int32_t *)p));
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
}
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (OSAtomicAdd32(-((int32_t)x), (int32_t *)p));
}
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (!OSAtomicCompareAndSwap32(c, s, (int32_t *)p));
Implement atomic operations for x86/x64. Add inline assembly implementations of atomic_{add,sub}_uint{32,64}() for x86/x64, in order to support compilers that are missing the relevant gcc intrinsics.
2011-03-25 07:48:11 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint32_t o;
/*The documented OSAtomic*() API does not expose an atomic exchange. */
do {
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
o = atomic_read_u32(p);
} while (atomic_cas_u32(p, o, x));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
}
#elif (defined(_MSC_VER))
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (InterlockedExchangeAdd(p, x) + x);
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
}
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (InterlockedExchangeAdd(p, -((int32_t)x)) - x);
}
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
uint32_t o;
Use InterlockedCompareExchange instead of non-existing InterlockedCompareExchange32
2015-03-17 11:09:30 +08:00
o = InterlockedCompareExchange(p, s, c);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (o != c);
Add atomic(9) implementations of atomic operations. Add atomic(9) implementations of atomic operations. These are used on FreeBSD for non-x86 architectures.
2012-04-18 04:17:54 +08:00
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
InterlockedExchange(p, x);
}
#elif (defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_4) || \
defined(JE_FORCE_SYNC_COMPARE_AND_SWAP_4))
Add SH4 and mips architecture support. Submitted by Andreas Vinsander.
2012-03-06 04:16:57 +08:00
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u32(uint32_t *p, uint32_t x) {
Add SH4 and mips architecture support. Submitted by Andreas Vinsander.
2012-03-06 04:16:57 +08:00
return (__sync_add_and_fetch(p, x));
}
JEMALLOC_INLINE uint32_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u32(uint32_t *p, uint32_t x) {
Add SH4 and mips architecture support. Submitted by Andreas Vinsander.
2012-03-06 04:16:57 +08:00
return (__sync_sub_and_fetch(p, x));
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u32(uint32_t *p, uint32_t c, uint32_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
return (!__sync_bool_compare_and_swap(p, c, s));
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u32(uint32_t *p, uint32_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
__sync_lock_test_and_set(p, x);
}
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
#else
# error "Missing implementation for 32-bit atomic operations"
#endif
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
/******************************************************************************/
/* Pointer operations. */
JEMALLOC_INLINE void *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_p(void **p, void *x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((void *)atomic_add_u64((uint64_t *)p, (uint64_t)x));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((void *)atomic_add_u32((uint32_t *)p, (uint32_t)x));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
JEMALLOC_INLINE void *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_p(void **p, void *x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((void *)atomic_add_u64((uint64_t *)p, (uint64_t)-((int64_t)x)));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((void *)atomic_add_u32((uint32_t *)p, (uint32_t)-((int32_t)x)));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_p(void **p, void *c, void *s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return (atomic_cas_u64((uint64_t *)p, (uint64_t)c, (uint64_t)s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return (atomic_cas_u32((uint32_t *)p, (uint32_t)c, (uint32_t)s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_p(void **p, const void *x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
atomic_write_u64((uint64_t *)p, (uint64_t)x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
atomic_write_u32((uint32_t *)p, (uint32_t)x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
/******************************************************************************/
/* size_t operations. */
JEMALLOC_INLINE size_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_zu(size_t *p, size_t x) {
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((size_t)atomic_add_u64((uint64_t *)p, (uint64_t)x));
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((size_t)atomic_add_u32((uint32_t *)p, (uint32_t)x));
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
#endif
}
JEMALLOC_INLINE size_t
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_zu(size_t *p, size_t x) {
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((size_t)atomic_add_u64((uint64_t *)p, (uint64_t)-((int64_t)x)));
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((size_t)atomic_add_u32((uint32_t *)p, (uint32_t)-((int32_t)x)));
Restructure atomic_*_z(). Restructure atomic_*_z() so that no casting within macros is necessary. This avoids warnings when compiling with clang.
2012-03-24 07:09:56 +08:00
#endif
}
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_zu(size_t *p, size_t c, size_t s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return (atomic_cas_u64((uint64_t *)p, (uint64_t)c, (uint64_t)s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return (atomic_cas_u32((uint32_t *)p, (uint32_t)c, (uint32_t)s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_zu(size_t *p, size_t x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_PTR == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
atomic_write_u64((uint64_t *)p, (uint64_t)x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_PTR == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
atomic_write_u32((uint32_t *)p, (uint32_t)x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
/******************************************************************************/
/* unsigned operations. */
JEMALLOC_INLINE unsigned
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_add_u(unsigned *p, unsigned x) {
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
#if (LG_SIZEOF_INT == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((unsigned)atomic_add_u64((uint64_t *)p, (uint64_t)x));
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
#elif (LG_SIZEOF_INT == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((unsigned)atomic_add_u32((uint32_t *)p, (uint32_t)x));
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
#endif
}
JEMALLOC_INLINE unsigned
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_sub_u(unsigned *p, unsigned x) {
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
#if (LG_SIZEOF_INT == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((unsigned)atomic_add_u64((uint64_t *)p,
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
(uint64_t)-((int64_t)x)));
#elif (LG_SIZEOF_INT == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return ((unsigned)atomic_add_u32((uint32_t *)p,
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
(uint32_t)-((int32_t)x)));
#endif
}
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
JEMALLOC_INLINE bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_cas_u(unsigned *p, unsigned c, unsigned s) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_INT == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return (atomic_cas_u64((uint64_t *)p, (uint64_t)c, (uint64_t)s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_INT == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
return (atomic_cas_u32((uint32_t *)p, (uint32_t)c, (uint32_t)s));
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
JEMALLOC_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
atomic_write_u(unsigned *p, unsigned x) {
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#if (LG_SIZEOF_INT == 3)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
atomic_write_u64((uint64_t *)p, (uint64_t)x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#elif (LG_SIZEOF_INT == 2)
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
atomic_write_u32((uint32_t *)p, (uint32_t)x);
Implement more atomic operations. - atomic_*_p(). - atomic_cas_*(). - atomic_write_*().
2015-02-03 05:49:08 +08:00
#endif
}
Remove ephemeral mutexes. Remove ephemeral mutexes from the prof machinery, and remove malloc_mutex_destroy(). This simplifies mutex management on systems that call malloc()/free() inside pthread_mutex_{create,destroy}(). Add atomic_*_u() for operation on unsigned values. Fix prof_printf() to call malloc_vsnprintf() rather than malloc_snprintf().
2012-03-24 09:05:51 +08:00
/******************************************************************************/
Add atomic.[ch]. Add atomic.[ch], which should have been part of the previous commit.
2011-03-19 09:15:37 +08:00
#endif
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#endif /* JEMALLOC_INTERNAL_ATOMIC_INLINES_H */
Reference in New Issue Copy Permalink