project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
c1baa0a9b7
server-skynet-source-3rd-je.../src/extent.c

1165 lines
33 KiB
C
Raw Normal View History

Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#define JEMALLOC_EXTENT_C_
Restructure source tree.
2010-02-12 06:45:59 +08:00
#include "jemalloc/internal/jemalloc_internal.h"
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
/******************************************************************************/
/* Data. */
rtree_t extents_rtree;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
static void *extent_alloc_default(extent_hooks_t *extent_hooks,
void *new_addr, size_t size, size_t alignment, bool *zero, bool *commit,
unsigned arena_ind);
static bool extent_dalloc_default(extent_hooks_t *extent_hooks, void *addr,
size_t size, bool committed, unsigned arena_ind);
static bool extent_commit_default(extent_hooks_t *extent_hooks, void *addr,
size_t size, size_t offset, size_t length, unsigned arena_ind);
static bool extent_decommit_default(extent_hooks_t *extent_hooks,
void *addr, size_t size, size_t offset, size_t length, unsigned arena_ind);
static bool extent_purge_default(extent_hooks_t *extent_hooks, void *addr,
size_t size, size_t offset, size_t length, unsigned arena_ind);
static bool extent_split_default(extent_hooks_t *extent_hooks, void *addr,
size_t size, size_t size_a, size_t size_b, bool committed,
unsigned arena_ind);
static bool extent_merge_default(extent_hooks_t *extent_hooks, void *addr_a,
size_t size_a, void *addr_b, size_t size_b, bool committed,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
unsigned arena_ind);
const extent_hooks_t extent_hooks_default = {
extent_alloc_default,
extent_dalloc_default,
extent_commit_default,
extent_decommit_default,
extent_purge_default,
extent_split_default,
extent_merge_default
};
/* Used exclusively for gdump triggering. */
Fix gdump triggering regression. Now that extents are not multiples of chunksize, it's necessary to track pages rather than chunks.
2016-06-06 05:43:20 +08:00
static size_t curpages;
static size_t highpages;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/******************************************************************************/
/*
* Function prototypes for static functions that are referenced prior to
* definition.
*/
static void extent_record(tsdn_t *tsdn, arena_t *arena,
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_hooks_t **r_extent_hooks, extent_heap_t extent_heaps[NPSIZES+1],
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
bool cache, extent_t *extent);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
/******************************************************************************/
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
extent_t *
extent_alloc(tsdn_t *tsdn, arena_t *arena)
{
extent_t *extent;
malloc_mutex_lock(tsdn, &arena->extent_cache_mtx);
extent = ql_last(&arena->extent_cache, ql_link);
if (extent == NULL) {
malloc_mutex_unlock(tsdn, &arena->extent_cache_mtx);
return (base_alloc(tsdn, sizeof(extent_t)));
}
ql_tail_remove(&arena->extent_cache, extent_t, ql_link);
malloc_mutex_unlock(tsdn, &arena->extent_cache_mtx);
return (extent);
}
void
extent_dalloc(tsdn_t *tsdn, arena_t *arena, extent_t *extent)
{
malloc_mutex_lock(tsdn, &arena->extent_cache_mtx);
ql_elm_new(extent, ql_link);
ql_tail_insert(&arena->extent_cache, extent, ql_link);
malloc_mutex_unlock(tsdn, &arena->extent_cache_mtx);
}
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t *
extent_hooks_get(arena_t *arena)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
return ((extent_hooks_t *)atomic_read_p(&arena->extent_hooks_pun));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t *
extent_hooks_set(arena_t *arena, extent_hooks_t *extent_hooks)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t *old_extent_hooks = extent_hooks_get(arena);
union {
extent_hooks_t **h;
void **v;
} u;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
u.h = &arena->extent_hooks;
atomic_write_p(u.v, extent_hooks);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (old_extent_hooks);
}
static void
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena_t *arena, extent_hooks_t **r_extent_hooks)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
if (*r_extent_hooks == EXTENT_HOOKS_INITIALIZER)
*r_extent_hooks = extent_hooks_get(arena);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
#ifdef JEMALLOC_JET
#undef extent_size_quantize_floor
#define extent_size_quantize_floor JEMALLOC_N(n_extent_size_quantize_floor)
#endif
size_t
extent_size_quantize_floor(size_t size)
Implement cache index randomization for large allocations. Extract szad size quantization into {extent,run}_quantize(), and . quantize szad run sizes to the union of valid small region run sizes and large run sizes. Refactor iteration in arena_run_first_fit() to use run_quantize{,_first,_next(), and add support for padded large runs. For large allocations that have no specified alignment constraints, compute a pseudo-random offset from the beginning of the first backing page that is a multiple of the cache line size. Under typical configurations with 4-KiB pages and 64-byte cache lines this results in a uniform distribution among 64 page boundary offsets. Add the --disable-cache-oblivious option, primarily intended for performance testing. This resolves #13.
2015-05-05 00:58:36 +08:00
{
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
size_t ret;
pszind_t pind;
Implement cache index randomization for large allocations. Extract szad size quantization into {extent,run}_quantize(), and . quantize szad run sizes to the union of valid small region run sizes and large run sizes. Refactor iteration in arena_run_first_fit() to use run_quantize{,_first,_next(), and add support for padded large runs. For large allocations that have no specified alignment constraints, compute a pseudo-random offset from the beginning of the first backing page that is a multiple of the cache line size. Under typical configurations with 4-KiB pages and 64-byte cache lines this results in a uniform distribution among 64 page boundary offsets. Add the --disable-cache-oblivious option, primarily intended for performance testing. This resolves #13.
2015-05-05 00:58:36 +08:00
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
assert(size > 0);
assert((size & PAGE_MASK) == 0);
assert(size != 0);
assert(size == PAGE_CEILING(size));
pind = psz2ind(size - large_pad + 1);
if (pind == 0) {
/*
* Avoid underflow. This short-circuit would also do the right
* thing for all sizes in the range for which there are
* PAGE-spaced size classes, but it's simplest to just handle
* the one case that would cause erroneous results.
*/
return (size);
}
ret = pind2sz(pind - 1) + large_pad;
assert(ret <= size);
return (ret);
Implement cache index randomization for large allocations. Extract szad size quantization into {extent,run}_quantize(), and . quantize szad run sizes to the union of valid small region run sizes and large run sizes. Refactor iteration in arena_run_first_fit() to use run_quantize{,_first,_next(), and add support for padded large runs. For large allocations that have no specified alignment constraints, compute a pseudo-random offset from the beginning of the first backing page that is a multiple of the cache line size. Under typical configurations with 4-KiB pages and 64-byte cache lines this results in a uniform distribution among 64 page boundary offsets. Add the --disable-cache-oblivious option, primarily intended for performance testing. This resolves #13.
2015-05-05 00:58:36 +08:00
}
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
#ifdef JEMALLOC_JET
#undef extent_size_quantize_floor
#define extent_size_quantize_floor JEMALLOC_N(extent_size_quantize_floor)
extent_size_quantize_t *extent_size_quantize_floor =
JEMALLOC_N(n_extent_size_quantize_floor);
#endif
Implement cache index randomization for large allocations. Extract szad size quantization into {extent,run}_quantize(), and . quantize szad run sizes to the union of valid small region run sizes and large run sizes. Refactor iteration in arena_run_first_fit() to use run_quantize{,_first,_next(), and add support for padded large runs. For large allocations that have no specified alignment constraints, compute a pseudo-random offset from the beginning of the first backing page that is a multiple of the cache line size. Under typical configurations with 4-KiB pages and 64-byte cache lines this results in a uniform distribution among 64 page boundary offsets. Add the --disable-cache-oblivious option, primarily intended for performance testing. This resolves #13.
2015-05-05 00:58:36 +08:00
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
#ifdef JEMALLOC_JET
#undef extent_size_quantize_ceil
#define extent_size_quantize_ceil JEMALLOC_N(n_extent_size_quantize_ceil)
#endif
size_t
extent_size_quantize_ceil(size_t size)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
{
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
size_t ret;
assert(size > 0);
Rename huge to large.
2016-06-01 05:50:21 +08:00
assert(size - large_pad <= LARGE_MAXCLASS);
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
assert((size & PAGE_MASK) == 0);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
ret = extent_size_quantize_floor(size);
if (ret < size) {
/*
* Skip a quantization that may have an adequately large extent,
* because under-sized extents may be mixed in. This only
* happens when an unusual size is requested, i.e. for aligned
* allocation, and is just one of several places where linear
* search would potentially find sufficiently aligned available
* memory somewhere lower.
*/
ret = pind2sz(psz2ind(ret - large_pad + 1)) + large_pad;
}
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
return (ret);
}
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
#ifdef JEMALLOC_JET
#undef extent_size_quantize_ceil
#define extent_size_quantize_ceil JEMALLOC_N(extent_size_quantize_ceil)
extent_size_quantize_t *extent_size_quantize_ceil =
JEMALLOC_N(n_extent_size_quantize_ceil);
#endif
/* Generate pairing heap functions. */
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
ph_gen(, extent_heap_, extent_heap_t, extent_t, ph_link, extent_snad_comp)
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_heaps_insert(tsdn_t *tsdn, extent_heap_t extent_heaps[NPSIZES+1],
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
size_t psz = extent_size_quantize_floor(extent_size_get(extent));
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
pszind_t pind = psz2ind(psz);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
malloc_mutex_assert_owner(tsdn, &extent_arena_get(extent)->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_heap_insert(&extent_heaps[pind], extent);
}
static void
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_heaps_remove(tsdn_t *tsdn, extent_heap_t extent_heaps[NPSIZES+1],
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
size_t psz = extent_size_quantize_floor(extent_size_get(extent));
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
pszind_t pind = psz2ind(psz);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
malloc_mutex_assert_owner(tsdn, &extent_arena_get(extent)->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_heap_remove(&extent_heaps[pind], extent);
}
static bool
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_rtree_acquire(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx,
const extent_t *extent, bool dependent, bool init_missing,
rtree_elm_t **r_elm_a, rtree_elm_t **r_elm_b)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
*r_elm_a = rtree_elm_acquire(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_base_get(extent), dependent, init_missing);
if (!dependent && *r_elm_a == NULL)
return (true);
assert(*r_elm_a != NULL);
if (extent_size_get(extent) > PAGE) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
*r_elm_b = rtree_elm_acquire(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_last_get(extent), dependent,
init_missing);
Fix extent_rtree acquire() to release element on error. This resolves #480.
2016-11-01 07:32:33 +08:00
if (!dependent && *r_elm_b == NULL) {
rtree_elm_release(tsdn, &extents_rtree, *r_elm_a);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (true);
Fix extent_rtree acquire() to release element on error. This resolves #480.
2016-11-01 07:32:33 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(*r_elm_b != NULL);
} else
*r_elm_b = NULL;
return (false);
}
static void
extent_rtree_write_acquired(tsdn_t *tsdn, rtree_elm_t *elm_a,
rtree_elm_t *elm_b, const extent_t *extent)
{
rtree_elm_write_acquired(tsdn, &extents_rtree, elm_a, extent);
if (elm_b != NULL)
rtree_elm_write_acquired(tsdn, &extents_rtree, elm_b, extent);
}
static void
extent_rtree_release(tsdn_t *tsdn, rtree_elm_t *elm_a, rtree_elm_t *elm_b)
{
rtree_elm_release(tsdn, &extents_rtree, elm_a);
if (elm_b != NULL)
rtree_elm_release(tsdn, &extents_rtree, elm_b);
}
static void
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_register(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx,
const extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
size_t i;
assert(extent_slab_get(extent));
for (i = 1; i < (extent_size_get(extent) >> LG_PAGE) - 1; i++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_write(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_base_get(extent) + (uintptr_t)(i <<
LG_PAGE), extent);
}
}
static bool
extent_register(tsdn_t *tsdn, const extent_t *extent)
{
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
rtree_elm_t *elm_a, *elm_b;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
if (extent_rtree_acquire(tsdn, rtree_ctx, extent, false, true, &elm_a,
&elm_b))
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (true);
extent_rtree_write_acquired(tsdn, elm_a, elm_b, extent);
if (extent_slab_get(extent))
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_register(tsdn, rtree_ctx, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_rtree_release(tsdn, elm_a, elm_b);
if (config_prof && opt_prof && extent_active_get(extent)) {
Fix gdump triggering regression. Now that extents are not multiples of chunksize, it's necessary to track pages rather than chunks.
2016-06-06 05:43:20 +08:00
size_t nadd = extent_size_get(extent) >> LG_PAGE;
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
size_t cur = atomic_add_zu(&curpages, nadd);
size_t high = atomic_read_zu(&highpages);
while (cur > high && atomic_cas_zu(&highpages, high, cur)) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/*
* Don't refresh cur, because it may have decreased
Fix gdump triggering regression. Now that extents are not multiples of chunksize, it's necessary to track pages rather than chunks.
2016-06-06 05:43:20 +08:00
* since this thread lost the highpages update race.
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
*/
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
high = atomic_read_zu(&highpages);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
if (cur > high && prof_gdump_get_unlocked())
prof_gdump(tsdn);
}
return (false);
}
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
static void
extent_reregister(tsdn_t *tsdn, const extent_t *extent)
{
bool err = extent_register(tsdn, extent);
assert(!err);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_deregister(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx,
const extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
size_t i;
assert(extent_slab_get(extent));
for (i = 1; i < (extent_size_get(extent) >> LG_PAGE) - 1; i++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_clear(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_base_get(extent) + (uintptr_t)(i <<
LG_PAGE));
}
}
static void
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
extent_deregister(tsdn_t *tsdn, extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
rtree_elm_t *elm_a, *elm_b;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_rtree_acquire(tsdn, rtree_ctx, extent, true, false, &elm_a,
&elm_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_rtree_write_acquired(tsdn, elm_a, elm_b, NULL);
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
if (extent_slab_get(extent)) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_deregister(tsdn, rtree_ctx, extent);
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
extent_slab_set(extent, false);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_rtree_release(tsdn, elm_a, elm_b);
if (config_prof && opt_prof && extent_active_get(extent)) {
Fix gdump triggering regression. Now that extents are not multiples of chunksize, it's necessary to track pages rather than chunks.
2016-06-06 05:43:20 +08:00
size_t nsub = extent_size_get(extent) >> LG_PAGE;
Rename atomic_*_{uint32,uint64,u}() to atomic_*_{u32,u64,zu}(). This change conforms to naming conventions throughout the codebase.
2016-11-08 03:27:48 +08:00
assert(atomic_read_zu(&curpages) >= nsub);
atomic_sub_zu(&curpages, nsub);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
}
/*
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
* Do first-best-fit extent selection, i.e. select the oldest/lowest extent that
* best fits.
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
*/
static extent_t *
Add various mutex ownership assertions.
2016-09-24 03:18:36 +08:00
extent_first_best_fit(tsdn_t *tsdn, arena_t *arena,
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_heap_t extent_heaps[NPSIZES+1], size_t size)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
pszind_t pind, i;
Add various mutex ownership assertions.
2016-09-24 03:18:36 +08:00
malloc_mutex_assert_owner(tsdn, &arena->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
pind = psz2ind(extent_size_quantize_ceil(size));
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
for (i = pind; i < NPSIZES+1; i++) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_t *extent = extent_heap_first(&extent_heaps[i]);
if (extent != NULL)
return (extent);
}
return (NULL);
}
static void
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_leak(tsdn_t *tsdn, arena_t *arena, extent_hooks_t **r_extent_hooks,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
bool cache, extent_t *extent)
{
/*
* Leak extent after making sure its pages have already been purged, so
* that this is only a virtual memory leak.
*/
if (cache) {
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_purge_wrapper(tsdn, arena, r_extent_hooks, extent, 0,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_size_get(extent));
}
extent_dalloc(tsdn, arena, extent);
}
static extent_t *
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_recycle(tsdn_t *tsdn, arena_t *arena, extent_hooks_t **r_extent_hooks,
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_heap_t extent_heaps[NPSIZES+1], bool locked, bool cache,
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
void *new_addr, size_t usize, size_t pad, size_t alignment, bool *zero,
bool *commit, bool slab)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
extent_t *extent;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t size, alloc_size, leadsize, trailsize;
Add various mutex ownership assertions.
2016-09-24 03:18:36 +08:00
if (locked)
malloc_mutex_assert_owner(tsdn, &arena->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(new_addr == NULL || !slab);
assert(pad == 0 || !slab);
Fix/simplify extent_recycle() allocation size computations. Do not call s2u() during alloc_size computation, since any necessary ceiling increase is taken care of later by extent_first_best_fit() --> extent_size_quantize_ceil(), and the s2u() call may erroneously cause a higher quantization result. Remove an overly strict overflow check that was added in 4a7852137d8b6598fdb90ea8e1fd3bc8a8b94a3a (Fix extent_recycle()'s cache-oblivious padding support.).
2016-11-04 14:49:21 +08:00
assert(alignment > 0);
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
if (config_debug && new_addr != NULL) {
extent_t *prev;
/*
* Non-NULL new_addr has two use cases:
*
* 1) Recycle a known-extant extent, e.g. during purging.
* 2) Perform in-place expanding reallocation.
*
* Regardless of use case, new_addr must either refer to a
* non-existing extent, or to the base of an extant extent,
* since only active slabs support interior lookups (which of
* course cannot be recycled).
*/
assert(PAGE_ADDR2BASE(new_addr) == new_addr);
Fix size class overflow bugs. Avoid calling s2u() on raw extent sizes in extent_recycle(). Clamp psz2ind() (implemented as psz2ind_clamp()) when inserting/removing into/from size-segregated extent heaps.
2016-10-04 05:18:55 +08:00
assert(pad == 0);
assert(alignment <= PAGE);
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
prev = extent_lookup(tsdn, (void *)((uintptr_t)new_addr - PAGE),
false);
assert(prev == NULL || extent_past_get(prev) == new_addr);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Fix extent_recycle()'s cache-oblivious padding support. Add padding *after* computing the size class, so that the optimal size class isn't skipped during search for a usable extent. This regression was caused by b46261d58b449cc4c099ed2384451a2499688f0e (Implement cache-oblivious support for huge size classes.).
2016-11-04 12:14:59 +08:00
size = usize + pad;
Fix/simplify extent_recycle() allocation size computations. Do not call s2u() during alloc_size computation, since any necessary ceiling increase is taken care of later by extent_first_best_fit() --> extent_size_quantize_ceil(), and the s2u() call may erroneously cause a higher quantization result. Remove an overly strict overflow check that was added in 4a7852137d8b6598fdb90ea8e1fd3bc8a8b94a3a (Fix extent_recycle()'s cache-oblivious padding support.).
2016-11-04 14:49:21 +08:00
alloc_size = size + PAGE_CEILING(alignment) - PAGE;
/* Beware size_t wrap-around. */
if (alloc_size < usize)
return (NULL);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
if (!locked)
malloc_mutex_lock(tsdn, &arena->extents_mtx);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (new_addr != NULL) {
rtree_elm_t *elm;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
elm = rtree_elm_acquire(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)new_addr, false, false);
if (elm != NULL) {
extent = rtree_elm_read_acquired(tsdn, &extents_rtree,
elm);
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
if (extent != NULL) {
assert(extent_base_get(extent) == new_addr);
if (extent_arena_get(extent) != arena ||
extent_size_get(extent) < size ||
extent_active_get(extent) ||
extent_retained_get(extent) == cache)
extent = NULL;
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
rtree_elm_release(tsdn, &extents_rtree, elm);
} else
extent = NULL;
Add various mutex ownership assertions.
2016-09-24 03:18:36 +08:00
} else {
extent = extent_first_best_fit(tsdn, arena, extent_heaps,
alloc_size);
}
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
if (extent == NULL) {
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
if (!locked)
malloc_mutex_unlock(tsdn, &arena->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (NULL);
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_remove(tsdn, extent_heaps, extent);
arena_extent_cache_maybe_remove(tsdn, arena, extent, cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
leadsize = ALIGNMENT_CEILING((uintptr_t)extent_base_get(extent),
PAGE_CEILING(alignment)) - (uintptr_t)extent_base_get(extent);
assert(new_addr == NULL || leadsize == 0);
assert(extent_size_get(extent) >= leadsize + size);
trailsize = extent_size_get(extent) - leadsize - size;
if (extent_zeroed_get(extent))
*zero = true;
if (extent_committed_get(extent))
*commit = true;
/* Split the lead. */
if (leadsize != 0) {
extent_t *lead = extent;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent = extent_split_wrapper(tsdn, arena, r_extent_hooks,
lead, leadsize, leadsize, size + trailsize, usize +
trailsize);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (extent == NULL) {
Fix regressions related extent splitting failures. Fix a fundamental extent_split_wrapper() bug in an error path. Fix extent_recycle() to deregister unsplittable extents before leaking them. Relax xallocx() test assertions so that unsplittable extents don't cause test failures.
2016-06-06 13:08:20 +08:00
extent_deregister(tsdn, lead);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_leak(tsdn, arena, r_extent_hooks, cache, lead);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
if (!locked)
malloc_mutex_unlock(tsdn, &arena->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (NULL);
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_insert(tsdn, extent_heaps, lead);
arena_extent_cache_maybe_insert(tsdn, arena, lead, cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
/* Split the trail. */
if (trailsize != 0) {
extent_t *trail = extent_split_wrapper(tsdn, arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
r_extent_hooks, extent, size, usize, trailsize, trailsize);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (trail == NULL) {
Fix regressions related extent splitting failures. Fix a fundamental extent_split_wrapper() bug in an error path. Fix extent_recycle() to deregister unsplittable extents before leaking them. Relax xallocx() test assertions so that unsplittable extents don't cause test failures.
2016-06-06 13:08:20 +08:00
extent_deregister(tsdn, extent);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_leak(tsdn, arena, r_extent_hooks, cache,
extent);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
if (!locked)
malloc_mutex_unlock(tsdn, &arena->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (NULL);
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_insert(tsdn, extent_heaps, trail);
arena_extent_cache_maybe_insert(tsdn, arena, trail, cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
} else if (leadsize == 0) {
/*
* Splitting causes usize to be set as a side effect, but no
* splitting occurred.
*/
extent_usize_set(extent, usize);
}
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
if (commit && !extent_committed_get(extent) &&
extent_commit_wrapper(tsdn, arena, r_extent_hooks, extent, 0,
extent_size_get(extent))) {
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
if (!locked)
malloc_mutex_unlock(tsdn, &arena->extents_mtx);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_record(tsdn, arena, r_extent_hooks, extent_heaps, cache,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent);
return (NULL);
}
if (pad != 0)
extent_addr_randomize(tsdn, extent, alignment);
extent_active_set(extent, true);
if (slab) {
extent_slab_set(extent, slab);
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_register(tsdn, rtree_ctx, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
if (!locked)
malloc_mutex_unlock(tsdn, &arena->extents_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (*zero) {
if (!extent_zeroed_get(extent)) {
memset(extent_addr_get(extent), 0,
extent_usize_get(extent));
} else if (config_debug) {
size_t i;
size_t *p = (size_t *)(uintptr_t)
extent_addr_get(extent);
for (i = 0; i < usize / sizeof(size_t); i++)
assert(p[i] == 0);
}
}
return (extent);
}
/*
* If the caller specifies (!*zero), it is still possible to receive zeroed
* memory, in which case *zero is toggled to true. arena_extent_alloc() takes
* advantage of this to avoid demanding zeroed extents, but taking advantage of
* them if they are returned.
*/
static void *
extent_alloc_core(tsdn_t *tsdn, arena_t *arena, void *new_addr, size_t size,
size_t alignment, bool *zero, bool *commit, dss_prec_t dss_prec)
{
void *ret;
assert(size != 0);
assert(alignment != 0);
/* "primary" dss. */
if (have_dss && dss_prec == dss_prec_primary && (ret =
extent_alloc_dss(tsdn, arena, new_addr, size, alignment, zero,
commit)) != NULL)
return (ret);
/* mmap. */
if ((ret = extent_alloc_mmap(new_addr, size, alignment, zero, commit))
!= NULL)
return (ret);
/* "secondary" dss. */
if (have_dss && dss_prec == dss_prec_secondary && (ret =
extent_alloc_dss(tsdn, arena, new_addr, size, alignment, zero,
commit)) != NULL)
return (ret);
/* All strategies for allocation failed. */
return (NULL);
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
static extent_t *
extent_alloc_cache_impl(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, bool locked, void *new_addr, size_t usize,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
size_t pad, size_t alignment, bool *zero, bool *commit, bool slab)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
extent_t *extent;
assert(usize + pad != 0);
assert(alignment != 0);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent = extent_recycle(tsdn, arena, r_extent_hooks,
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
arena->extents_cached, locked, true, new_addr, usize, pad,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
alignment, zero, commit, slab);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (extent == NULL)
return (NULL);
return (extent);
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_t *
extent_alloc_cache_locked(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
size_t alignment, bool *zero, bool *commit, bool slab)
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
{
malloc_mutex_assert_owner(tsdn, &arena->extents_mtx);
return (extent_alloc_cache_impl(tsdn, arena, r_extent_hooks, true,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
new_addr, usize, pad, alignment, zero, commit, slab));
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
}
extent_t *
extent_alloc_cache(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
size_t alignment, bool *zero, bool *commit, bool slab)
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
{
return (extent_alloc_cache_impl(tsdn, arena, r_extent_hooks, false,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
new_addr, usize, pad, alignment, zero, commit, slab));
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
}
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
static void *
extent_alloc_default_impl(tsdn_t *tsdn, arena_t *arena, void *new_addr,
size_t size, size_t alignment, bool *zero, bool *commit)
{
void *ret;
ret = extent_alloc_core(tsdn, arena, new_addr, size, alignment, zero,
commit, arena->dss_prec);
if (ret == NULL)
return (NULL);
return (ret);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void *
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_alloc_default(extent_hooks_t *extent_hooks, void *new_addr, size_t size,
size_t alignment, bool *zero, bool *commit, unsigned arena_ind)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
tsdn_t *tsdn;
arena_t *arena;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
assert(extent_hooks == &extent_hooks_default);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
tsdn = tsdn_fetch();
arena = arena_get(tsdn, arena_ind, false);
/*
* The arena we're allocating on behalf of must have been initialized
* already.
*/
assert(arena != NULL);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
return (extent_alloc_default_impl(tsdn, arena, new_addr, size,
alignment, zero, commit));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static extent_t *
extent_alloc_retained(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t alignment, bool *zero, bool *commit, bool slab)
{
extent_t *extent;
assert(usize != 0);
assert(alignment != 0);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent = extent_recycle(tsdn, arena, r_extent_hooks,
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
arena->extents_retained, false, false, new_addr, usize, pad,
alignment, zero, commit, slab);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (extent != NULL && config_stats) {
size_t size = usize + pad;
arena->stats.retained -= size;
}
return (extent);
}
static extent_t *
extent_alloc_wrapper_hard(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t alignment, bool *zero, bool *commit, bool slab)
{
extent_t *extent;
size_t size;
void *addr;
size = usize + pad;
extent = extent_alloc(tsdn, arena);
if (extent == NULL)
return (NULL);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
if (*r_extent_hooks == &extent_hooks_default) {
/* Call directly to propagate tsdn. */
addr = extent_alloc_default_impl(tsdn, arena, new_addr, size,
alignment, zero, commit);
} else {
addr = (*r_extent_hooks)->alloc(*r_extent_hooks, new_addr, size,
alignment, zero, commit, arena->ind);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (addr == NULL) {
extent_dalloc(tsdn, arena, extent);
return (NULL);
}
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
extent_init(extent, arena, addr, size, usize,
arena_extent_sn_next(arena), true, zero, commit, slab);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (pad != 0)
extent_addr_randomize(tsdn, extent, alignment);
if (extent_register(tsdn, extent)) {
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_leak(tsdn, arena, r_extent_hooks, false, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (NULL);
}
return (extent);
}
extent_t *
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_alloc_wrapper(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
size_t alignment, bool *zero, bool *commit, bool slab)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
extent_t *extent;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent = extent_alloc_retained(tsdn, arena, r_extent_hooks, new_addr,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
usize, pad, alignment, zero, commit, slab);
if (extent == NULL) {
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent = extent_alloc_wrapper_hard(tsdn, arena, r_extent_hooks,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
new_addr, usize, pad, alignment, zero, commit, slab);
}
return (extent);
}
static bool
extent_can_coalesce(const extent_t *a, const extent_t *b)
{
if (extent_arena_get(a) != extent_arena_get(b))
return (false);
if (extent_active_get(a) != extent_active_get(b))
return (false);
if (extent_committed_get(a) != extent_committed_get(b))
return (false);
if (extent_retained_get(a) != extent_retained_get(b))
return (false);
return (true);
}
static void
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_try_coalesce(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *a, extent_t *b,
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_heap_t extent_heaps[NPSIZES+1], bool cache)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
if (!extent_can_coalesce(a, b))
return;
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_remove(tsdn, extent_heaps, a);
extent_heaps_remove(tsdn, extent_heaps, b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
arena_extent_cache_maybe_remove(tsdn, extent_arena_get(a), a, cache);
arena_extent_cache_maybe_remove(tsdn, extent_arena_get(b), b, cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
if (extent_merge_wrapper(tsdn, arena, r_extent_hooks, a, b)) {
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_insert(tsdn, extent_heaps, a);
extent_heaps_insert(tsdn, extent_heaps, b);
arena_extent_cache_maybe_insert(tsdn, extent_arena_get(a), a,
cache);
arena_extent_cache_maybe_insert(tsdn, extent_arena_get(b), b,
cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return;
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_insert(tsdn, extent_heaps, a);
arena_extent_cache_maybe_insert(tsdn, extent_arena_get(a), a, cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static void
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_record(tsdn_t *tsdn, arena_t *arena, extent_hooks_t **r_extent_hooks,
Fix psz/pind edge cases. Add an "over-size" extent heap in which to store extents which exceed the maximum size class (plus cache-oblivious padding, if enabled). Remove psz2ind_clamp() and use psz2ind() instead so that trying to allocate the maximum size class can in principle succeed. In practice, this allows assertions to hold so that OOM errors can be successfully generated.
2016-11-04 12:18:50 +08:00
extent_heap_t extent_heaps[NPSIZES+1], bool cache, extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
extent_t *prev, *next;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(!cache || !extent_zeroed_get(extent));
malloc_mutex_lock(tsdn, &arena->extents_mtx);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_usize_set(extent, 0);
extent_active_set(extent, false);
extent_zeroed_set(extent, !cache && extent_zeroed_get(extent));
if (extent_slab_get(extent)) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_deregister(tsdn, rtree_ctx, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_slab_set(extent, false);
}
assert(extent_lookup(tsdn, extent_base_get(extent), true) == extent);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_heaps_insert(tsdn, extent_heaps, extent);
arena_extent_cache_maybe_insert(tsdn, arena, extent, cache);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* Try to coalesce forward. */
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
next = rtree_read(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_past_get(extent), false);
if (next != NULL) {
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_try_coalesce(tsdn, arena, r_extent_hooks, extent, next,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_heaps, cache);
}
/* Try to coalesce backward. */
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
prev = rtree_read(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_before_get(extent), false);
if (prev != NULL) {
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_try_coalesce(tsdn, arena, r_extent_hooks, prev, extent,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_heaps, cache);
}
malloc_mutex_unlock(tsdn, &arena->extents_mtx);
}
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
void
extent_dalloc_gap(tsdn_t *tsdn, arena_t *arena, extent_t *extent)
{
extent_hooks_t *extent_hooks = EXTENT_HOOKS_INITIALIZER;
if (extent_register(tsdn, extent)) {
extent_leak(tsdn, arena, &extent_hooks, false, extent);
return;
}
extent_dalloc_wrapper(tsdn, arena, &extent_hooks, extent);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
void
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_dalloc_cache(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
assert(extent_base_get(extent) != NULL);
assert(extent_size_get(extent) != 0);
extent_addr_set(extent, extent_base_get(extent));
extent_zeroed_set(extent, false);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_record(tsdn, arena, r_extent_hooks, arena->extents_cached, true,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent);
}
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
static bool
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
extent_dalloc_default_impl(void *addr, size_t size)
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
{
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
if (!have_dss || !extent_in_dss(addr))
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
return (extent_dalloc_mmap(addr, size));
return (true);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_dalloc_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
bool committed, unsigned arena_ind)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
assert(extent_hooks == &extent_hooks_default);
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
return (extent_dalloc_default_impl(addr, size));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
void
extent_dalloc_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
bool err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(extent_base_get(extent) != NULL);
assert(extent_size_get(extent) != 0);
extent_addr_set(extent, extent_base_get(extent));
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
/*
* Try to deallocate. Deregister first to avoid a race with other
* allocating threads, and reregister if deallocation fails.
*/
extent_deregister(tsdn, extent);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
if (*r_extent_hooks == &extent_hooks_default) {
/* Call directly to propagate tsdn. */
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
err = extent_dalloc_default_impl(extent_base_get(extent),
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
extent_size_get(extent));
} else {
err = (*r_extent_hooks)->dalloc(*r_extent_hooks,
extent_base_get(extent), extent_size_get(extent),
extent_committed_get(extent), arena->ind);
}
if (!err) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_dalloc(tsdn, arena, extent);
return;
}
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
extent_reregister(tsdn, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* Try to decommit; purge if that fails. */
if (extent_committed_get(extent)) {
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
extent_decommit_wrapper(tsdn, arena, r_extent_hooks, extent,
0, extent_size_get(extent));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
extent_zeroed_set(extent, !extent_committed_get(extent) ||
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
!(*r_extent_hooks)->purge(*r_extent_hooks, extent_base_get(extent),
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_size_get(extent), 0, extent_size_get(extent), arena->ind));
if (config_stats)
arena->stats.retained += extent_size_get(extent);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_record(tsdn, arena, r_extent_hooks, arena->extents_retained,
false, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_commit_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
size_t offset, size_t length, unsigned arena_ind)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
assert(extent_hooks == &extent_hooks_default);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (pages_commit((void *)((uintptr_t)addr + (uintptr_t)offset),
length));
}
bool
extent_commit_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t length)
{
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
bool err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
err = (*r_extent_hooks)->commit(*r_extent_hooks,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_base_get(extent), extent_size_get(extent), offset, length,
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
arena->ind);
extent_committed_set(extent, extent_committed_get(extent) || !err);
return (err);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_decommit_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
size_t offset, size_t length, unsigned arena_ind)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
assert(extent_hooks == &extent_hooks_default);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (pages_decommit((void *)((uintptr_t)addr + (uintptr_t)offset),
length));
}
bool
extent_decommit_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t length)
{
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
bool err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
err = (*r_extent_hooks)->decommit(*r_extent_hooks,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_base_get(extent), extent_size_get(extent), offset, length,
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
arena->ind);
extent_committed_set(extent, extent_committed_get(extent) && err);
return (err);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_purge_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
size_t offset, size_t length, unsigned arena_ind)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
assert(extent_hooks == &extent_hooks_default);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(addr != NULL);
assert((offset & PAGE_MASK) == 0);
assert(length != 0);
assert((length & PAGE_MASK) == 0);
return (pages_purge((void *)((uintptr_t)addr + (uintptr_t)offset),
length));
}
bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_purge_wrapper(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
size_t length)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
return ((*r_extent_hooks)->purge(*r_extent_hooks,
extent_base_get(extent), extent_size_get(extent), offset, length,
arena->ind));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_split_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
size_t size_a, size_t size_b, bool committed, unsigned arena_ind)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
assert(extent_hooks == &extent_hooks_default);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (!maps_coalesce)
return (true);
return (false);
}
extent_t *
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_split_wrapper(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t size_a,
size_t usize_a, size_t size_b, size_t usize_b)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
extent_t *trail;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
rtree_elm_t *lead_elm_a, *lead_elm_b, *trail_elm_a, *trail_elm_b;
assert(extent_size_get(extent) == size_a + size_b);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
trail = extent_alloc(tsdn, arena);
if (trail == NULL)
goto label_error_a;
{
extent_t lead;
extent_init(&lead, arena, extent_addr_get(extent), size_a,
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
usize_a, extent_sn_get(extent), extent_active_get(extent),
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_zeroed_get(extent), extent_committed_get(extent),
extent_slab_get(extent));
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
if (extent_rtree_acquire(tsdn, rtree_ctx, &lead, false, true,
&lead_elm_a, &lead_elm_b))
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
goto label_error_b;
}
extent_init(trail, arena, (void *)((uintptr_t)extent_base_get(extent) +
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
size_a), size_b, usize_b, extent_sn_get(extent),
extent_active_get(extent), extent_zeroed_get(extent),
extent_committed_get(extent), extent_slab_get(extent));
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
if (extent_rtree_acquire(tsdn, rtree_ctx, trail, false, true,
&trail_elm_a, &trail_elm_b))
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
goto label_error_c;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
if ((*r_extent_hooks)->split(*r_extent_hooks, extent_base_get(extent),
size_a + size_b, size_a, size_b, extent_committed_get(extent),
arena->ind))
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
goto label_error_d;
extent_size_set(extent, size_a);
extent_usize_set(extent, usize_a);
extent_rtree_write_acquired(tsdn, lead_elm_a, lead_elm_b, extent);
extent_rtree_write_acquired(tsdn, trail_elm_a, trail_elm_b, trail);
extent_rtree_release(tsdn, lead_elm_a, lead_elm_b);
extent_rtree_release(tsdn, trail_elm_a, trail_elm_b);
return (trail);
label_error_d:
Fix regressions related extent splitting failures. Fix a fundamental extent_split_wrapper() bug in an error path. Fix extent_recycle() to deregister unsplittable extents before leaking them. Relax xallocx() test assertions so that unsplittable extents don't cause test failures.
2016-06-06 13:08:20 +08:00
extent_rtree_release(tsdn, trail_elm_a, trail_elm_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
label_error_c:
extent_rtree_release(tsdn, lead_elm_a, lead_elm_b);
label_error_b:
extent_dalloc(tsdn, arena, trail);
label_error_a:
return (NULL);
}
static bool
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
extent_merge_default_impl(void *addr_a, void *addr_b)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
if (!maps_coalesce)
return (true);
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
if (have_dss && !extent_dss_mergeable(addr_a, addr_b))
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
return (true);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (false);
}
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
static bool
extent_merge_default(extent_hooks_t *extent_hooks, void *addr_a, size_t size_a,
void *addr_b, size_t size_b, bool committed, unsigned arena_ind)
{
assert(extent_hooks == &extent_hooks_default);
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
return (extent_merge_default_impl(addr_a, addr_b));
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_merge_wrapper(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *a, extent_t *b)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
bool err;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
rtree_elm_t *a_elm_a, *a_elm_b, *b_elm_a, *b_elm_b;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_assure_initialized(arena, r_extent_hooks);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
if (*r_extent_hooks == &extent_hooks_default) {
/* Call directly to propagate tsdn. */
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
err = extent_merge_default_impl(extent_base_get(a),
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
extent_base_get(b));
} else {
err = (*r_extent_hooks)->merge(*r_extent_hooks,
extent_base_get(a), extent_size_get(a), extent_base_get(b),
extent_size_get(b), extent_committed_get(a), arena->ind);
}
if (err)
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
return (true);
/*
* The rtree writes must happen while all the relevant elements are
* owned, so the following code uses decomposed helper functions rather
* than extent_{,de}register() to do things in the right order.
*/
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_rtree_acquire(tsdn, rtree_ctx, a, true, false, &a_elm_a,
&a_elm_b);
extent_rtree_acquire(tsdn, rtree_ctx, b, true, false, &b_elm_a,
&b_elm_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (a_elm_b != NULL) {
rtree_elm_write_acquired(tsdn, &extents_rtree, a_elm_b, NULL);
rtree_elm_release(tsdn, &extents_rtree, a_elm_b);
}
if (b_elm_b != NULL) {
rtree_elm_write_acquired(tsdn, &extents_rtree, b_elm_a, NULL);
rtree_elm_release(tsdn, &extents_rtree, b_elm_a);
} else
b_elm_b = b_elm_a;
extent_size_set(a, extent_size_get(a) + extent_size_get(b));
extent_usize_set(a, extent_usize_get(a) + extent_usize_get(b));
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
extent_sn_set(a, (extent_sn_get(a) < extent_sn_get(b)) ?
extent_sn_get(a) : extent_sn_get(b));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_zeroed_set(a, extent_zeroed_get(a) && extent_zeroed_get(b));
extent_rtree_write_acquired(tsdn, a_elm_a, b_elm_b, a);
extent_rtree_release(tsdn, a_elm_a, b_elm_b);
extent_dalloc(tsdn, extent_arena_get(b), b);
return (false);
}
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
bool
extent_boot(void)
{
if (rtree_new(&extents_rtree, (unsigned)((ZU(1) << (LG_SIZEOF_PTR+3)) -
LG_PAGE)))
return (true);
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
if (have_dss)
extent_dss_boot();
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
return (false);
}
Reference in New Issue Copy Permalink