project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ba8b9ecbcb
server-skynet-source-3rd-je.../src/extent.c

2091 lines
62 KiB
C
Raw Normal View History

Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define JEMALLOC_EXTENT_C_
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
#include "jemalloc/internal/jemalloc_preamble.h"
Header refactoring: move assert.h out of the catch-all
2017-04-12 05:43:12 +08:00
#include "jemalloc/internal/jemalloc_internal_includes.h"
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Header refactoring: move assert.h out of the catch-all
2017-04-12 05:43:12 +08:00
#include "jemalloc/internal/assert.h"
Header refactoring: unify and de-catchall extent_dss.
2017-05-24 05:36:09 +08:00
#include "jemalloc/internal/extent_dss.h"
Header refactoring: unify and de-catchall extent_mmap module.
2017-05-24 05:42:32 +08:00
#include "jemalloc/internal/extent_mmap.h"
Header refactoring: break out ph.h dependencies
2017-04-11 07:54:25 +08:00
#include "jemalloc/internal/ph.h"
Header refactoring: unify and de-catchall rtree module.
2017-05-24 05:26:31 +08:00
#include "jemalloc/internal/rtree.h"
Header refactoring: unify and de-catchall mutex module
2017-05-24 03:28:19 +08:00
#include "jemalloc/internal/mutex.h"
Header refactoring: unify and de-catchall mutex_pool.
2017-05-24 05:56:24 +08:00
#include "jemalloc/internal/mutex_pool.h"
Header refactoring: break out ph.h dependencies
2017-04-11 07:54:25 +08:00
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
/******************************************************************************/
/* Data. */
rtree_t extents_rtree;
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
/* Keyed by the address of the extent_t being protected. */
mutex_pool_t extent_mutex_pool;
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
Add opt.lg_extent_max_active_fit When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
2017-11-10 05:51:39 +08:00
size_t opt_lg_extent_max_active_fit = LG_EXTENT_MAX_ACTIVE_FIT_DEFAULT;
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static void *extent_alloc_default(extent_hooks_t *extent_hooks, void *new_addr,
size_t size, size_t alignment, bool *zero, bool *commit,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
unsigned arena_ind);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_dalloc_default(extent_hooks_t *extent_hooks, void *addr,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
size_t size, bool committed, unsigned arena_ind);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static void extent_destroy_default(extent_hooks_t *extent_hooks, void *addr,
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
size_t size, bool committed, unsigned arena_ind);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_commit_default(extent_hooks_t *extent_hooks, void *addr,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
size_t size, size_t offset, size_t length, unsigned arena_ind);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_commit_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length, bool growing_retained);
static bool extent_decommit_default(extent_hooks_t *extent_hooks,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
void *addr, size_t size, size_t offset, size_t length, unsigned arena_ind);
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
#ifdef PAGES_CAN_PURGE_LAZY
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_purge_lazy_default(extent_hooks_t *extent_hooks, void *addr,
size_t size, size_t offset, size_t length, unsigned arena_ind);
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
#endif
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_purge_lazy_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length, bool growing_retained);
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
#ifdef PAGES_CAN_PURGE_FORCED
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_purge_forced_default(extent_hooks_t *extent_hooks,
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
void *addr, size_t size, size_t offset, size_t length, unsigned arena_ind);
#endif
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_purge_forced_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length, bool growing_retained);
static bool extent_split_default(extent_hooks_t *extent_hooks, void *addr,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
size_t size, size_t size_a, size_t size_b, bool committed,
unsigned arena_ind);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static extent_t *extent_split_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t size_a,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
szind_t szind_a, bool slab_a, size_t size_b, szind_t szind_b, bool slab_b,
bool growing_retained);
static bool extent_merge_default(extent_hooks_t *extent_hooks, void *addr_a,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
size_t size_a, void *addr_b, size_t size_b, bool committed,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
unsigned arena_ind);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool extent_merge_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *a, extent_t *b,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool growing_retained);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
const extent_hooks_t extent_hooks_default = {
extent_alloc_default,
extent_dalloc_default,
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
extent_destroy_default,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_commit_default,
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
extent_decommit_default
#ifdef PAGES_CAN_PURGE_LAZY
,
extent_purge_lazy_default
#else
,
NULL
#endif
#ifdef PAGES_CAN_PURGE_FORCED
,
extent_purge_forced_default
#else
,
NULL
#endif
,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_split_default,
extent_merge_default
};
/* Used exclusively for gdump triggering. */
Convert extent module to use C11-style atomcis
2017-04-05 09:20:42 +08:00
static atomic_zu_t curpages;
static atomic_zu_t highpages;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/******************************************************************************/
/*
* Function prototypes for static functions that are referenced prior to
* definition.
*/
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
static void extent_deregister(tsdn_t *tsdn, extent_t *extent);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
static extent_t *extent_recycle(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, eset_t *eset, void *new_addr,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t usize, size_t pad, size_t alignment, bool slab, szind_t szind,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool *zero, bool *commit, bool growing_retained);
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
static extent_t *extent_try_coalesce(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
extent_t *extent, bool *coalesced, bool growing_retained);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
static void extent_record(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, eset_t *eset, extent_t *extent,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool growing_retained);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
/******************************************************************************/
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
#define ATTR_NONE /* does nothing */
ph_gen(ATTR_NONE, extent_avail_, extent_tree_t, extent_t, ph_link,
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
extent_esnead_comp)
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
#undef ATTR_NONE
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
typedef enum {
lock_result_success,
lock_result_failure,
lock_result_no_extent
} lock_result_t;
static lock_result_t
extent_rtree_leaf_elm_try_lock(tsdn_t *tsdn, rtree_leaf_elm_t *elm,
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_t **result, bool inactive_only) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_t *extent1 = rtree_leaf_elm_extent_read(tsdn, &extents_rtree,
elm, true);
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
/* Slab implies active extents and should be skipped. */
if (extent1 == NULL || (inactive_only && rtree_leaf_elm_slab_read(tsdn,
&extents_rtree, elm, true))) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
return lock_result_no_extent;
}
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
/*
* It's possible that the extent changed out from under us, and with it
* the leaf->extent mapping. We have to recheck while holding the lock.
*/
extent_lock(tsdn, extent1);
extent_t *extent2 = rtree_leaf_elm_extent_read(tsdn,
&extents_rtree, elm, true);
if (extent1 == extent2) {
*result = extent1;
return lock_result_success;
} else {
extent_unlock(tsdn, extent1);
return lock_result_failure;
}
}
/*
* Returns a pool-locked extent_t * if there's one associated with the given
* address, and NULL otherwise.
*/
static extent_t *
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_lock_from_addr(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx, void *addr,
bool inactive_only) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_t *ret = NULL;
rtree_leaf_elm_t *elm = rtree_leaf_elm_lookup(tsdn, &extents_rtree,
rtree_ctx, (uintptr_t)addr, false, false);
if (elm == NULL) {
return NULL;
}
lock_result_t lock_result;
do {
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
lock_result = extent_rtree_leaf_elm_try_lock(tsdn, elm, &ret,
inactive_only);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
} while (lock_result == lock_result_failure);
return ret;
}
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
static void
extent_addr_randomize(tsdn_t *tsdn, arena_t *arena, extent_t *extent,
size_t alignment) {
assert(extent_base_get(extent) == extent_addr_get(extent));
if (alignment < PAGE) {
unsigned lg_range = LG_PAGE -
lg_floor(CACHELINE_CEILING(alignment));
size_t r;
if (!tsdn_null(tsdn)) {
tsd_t *tsd = tsdn_tsd(tsdn);
r = (size_t)prng_lg_range_u64(
Rename tsd->offset_state to tsd->prng_state.
2019-11-05 08:48:12 +08:00
tsd_prng_statep_get(tsd), lg_range);
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
} else {
Avoid arena->offset_state when tsd not available for prng. Use stack locals and remove the offset_state in arena.
2019-11-05 08:44:37 +08:00
uint64_t stack_value = (uint64_t)(uintptr_t)&r;
r = (size_t)prng_lg_range_u64(&stack_value, lg_range);
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
}
uintptr_t random_offset = ((uintptr_t)r) << (LG_PAGE -
lg_range);
extent->e_addr = (void *)((uintptr_t)extent->e_addr +
random_offset);
assert(ALIGNMENT_ADDR2BASE(extent->e_addr, alignment) ==
extent->e_addr);
}
}
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
extent_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_alloc(tsdn_t *tsdn, arena_t *arena) {
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
malloc_mutex_lock(tsdn, &arena->extent_avail_mtx);
extent_t *extent = extent_avail_first(&arena->extent_avail);
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
if (extent == NULL) {
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
malloc_mutex_unlock(tsdn, &arena->extent_avail_mtx);
Track extent structure serial number (esn) in extent_t. This enables stable sorting of extent_t structures.
2017-04-17 12:51:26 +08:00
return base_alloc_extent(tsdn, arena->base);
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
}
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
extent_avail_remove(&arena->extent_avail, extent);
Add stats for the size of extent_avail heap
2018-08-02 05:14:33 +08:00
atomic_fetch_sub_zu(&arena->extent_avail_cnt, 1, ATOMIC_RELAXED);
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
malloc_mutex_unlock(tsdn, &arena->extent_avail_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent;
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
}
void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_dalloc(tsdn_t *tsdn, arena_t *arena, extent_t *extent) {
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
malloc_mutex_lock(tsdn, &arena->extent_avail_mtx);
extent_avail_insert(&arena->extent_avail, extent);
Add stats for the size of extent_avail heap
2018-08-02 05:14:33 +08:00
atomic_fetch_add_zu(&arena->extent_avail_cnt, 1, ATOMIC_RELAXED);
Prefer old/low extent_t structures during reuse. Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
2017-04-17 13:31:16 +08:00
malloc_mutex_unlock(tsdn, &arena->extent_avail_mtx);
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
}
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
ph_gen(, extent_heap_, extent_heap_t, extent_t, ph_link, extent_snad_comp)
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
static bool
extent_try_delayed_coalesce(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
extent_t *extent) {
extent_state_set(extent, extent_state_active);
bool coalesced;
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent = extent_try_coalesce(tsdn, arena, extent_hooks, rtree_ctx,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset, extent, &coalesced, false);
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
extent_state_set(extent, eset_state_get(eset));
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
if (!coalesced) {
return true;
}
Extent -> Eset: Move insertion and removal.
2019-09-21 14:51:13 +08:00
eset_insert_locked(tsdn, eset, extent);
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
return false;
}
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
extent_t *
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extents_alloc(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, void *new_addr, size_t size, size_t pad,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t alignment, bool slab, szind_t szind, bool *zero, bool *commit) {
assert(size + pad != 0);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
assert(alignment != 0);
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_t *extent = extent_recycle(tsdn, arena, extent_hooks, eset,
Add a "dumpable" bit to the extent state. Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
2017-09-19 08:25:57 +08:00
new_addr, size, pad, alignment, slab, szind, zero, commit, false);
assert(extent == NULL || extent_dumpable_get(extent));
return extent;
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
}
void
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extents_dalloc(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, extent_t *extent) {
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
assert(extent_base_get(extent) != NULL);
assert(extent_size_get(extent) != 0);
Add a "dumpable" bit to the extent state. Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
2017-09-19 08:25:57 +08:00
assert(extent_dumpable_get(extent));
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
extent_addr_set(extent, extent_base_get(extent));
extent_zeroed_set(extent, false);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks, eset, extent, false);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
}
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_t *
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extents_evict(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, size_t npages_min) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_lock(tsdn, &eset->mtx);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
/*
* Get the LRU coalesced extent, if any. If coalescing was delayed,
* the loop will iterate until the LRU extent is fully coalesced.
*/
extent_t *extent;
while (true) {
/* Get the LRU extent, if any. */
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent = extent_list_first(&eset->lru);
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
if (extent == NULL) {
goto label_return;
}
/* Check the eviction limit. */
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
size_t extents_npages = atomic_load_zu(&eset->npages,
Convert extents_t's npages field to use C11-style atomics In the process, we can do some strength reduction, changing the fetch-adds and fetch-subs to be simple loads followed by stores, since the modifications all occur while holding the mutex.
2017-03-08 09:57:48 +08:00
ATOMIC_RELAXED);
Over purge by 1 extent always. When purging, large allocations are usually the ones that cross the npages_limit threshold, simply because they are "large". This means we often leave the large extent around for a while, which has the downsides of: 1) high RSS and 2) more chance of them getting fragmented. Given that they are not likely to be reused very soon (LRU), let's over purge by 1 extent (which is often large and not reused frequently).
2017-12-09 04:13:50 +08:00
if (extents_npages <= npages_min) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
extent = NULL;
goto label_return;
}
Extent -> Eset: Move insertion and removal.
2019-09-21 14:51:13 +08:00
eset_remove_locked(tsdn, eset, extent);
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
if (!eset->delay_coalesce) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
break;
}
/* Try to coalesce. */
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_try_delayed_coalesce(tsdn, arena, extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
rtree_ctx, eset, extent)) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
break;
}
/*
* The LRU extent was just coalesced and the result placed in
* the LRU at its neighbor's position. Start over.
*/
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
/*
* Either mark the extent active or deregister it to protect against
* concurrent operations.
*/
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
switch (eset_state_get(eset)) {
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
case extent_state_active:
not_reached();
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
case extent_state_dirty:
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
case extent_state_muzzy:
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_state_set(extent, extent_state_active);
break;
case extent_state_retained:
extent_deregister(tsdn, extent);
break;
default:
not_reached();
}
label_return:
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_unlock(tsdn, &eset->mtx);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
return extent;
}
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
/*
* This can only happen when we fail to allocate a new extent struct (which
* indicates OOM), e.g. when trying to split an existing extent.
*/
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
static void
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extents_abandon_vm(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, extent_t *extent, bool growing_retained) {
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
size_t sz = extent_size_get(extent);
if (config_stats) {
arena_stats_accum_zu(&arena->stats.abandoned_vm, sz);
}
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/*
* Leak extent after making sure its pages have already been purged, so
* that this is only a virtual memory leak.
*/
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
if (eset_state_get(eset) == extent_state_dirty) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_purge_lazy_impl(tsdn, arena, extent_hooks,
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
extent, 0, sz, growing_retained)) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_purge_forced_impl(tsdn, arena, extent_hooks,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
extent, 0, extent_size_get(extent),
growing_retained);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
}
extent_dalloc(tsdn, arena, extent);
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate_locked(tsdn_t *tsdn, arena_t *arena, eset_t *eset,
extents: Remove preserve_lru feature. preserve_lru feature adds lots of complication, for little value. Removing it means merged extents are re-added to the lru list, and may take longer to madvise away than they otherwise would. Canaries after removal seem flat for several services (no change).
2018-02-23 01:40:27 +08:00
extent_t *extent) {
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
assert(extent_arena_ind_get(extent) == arena_ind_get(arena));
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
assert(extent_state_get(extent) == extent_state_active);
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
extent_state_set(extent, eset_state_get(eset));
Extent -> Eset: Move insertion and removal.
2019-09-21 14:51:13 +08:00
eset_insert_locked(tsdn, eset, extent);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
static void
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate(tsdn_t *tsdn, arena_t *arena, eset_t *eset,
extents: Remove preserve_lru feature. preserve_lru feature adds lots of complication, for little value. Removing it means merged extents are re-added to the lru list, and may take longer to madvise away than they otherwise would. Canaries after removal seem flat for several services (no change).
2018-02-23 01:40:27 +08:00
extent_t *extent) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_lock(tsdn, &eset->mtx);
extent_deactivate_locked(tsdn, arena, eset, extent);
malloc_mutex_unlock(tsdn, &eset->mtx);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
static void
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_activate_locked(tsdn_t *tsdn, arena_t *arena, eset_t *eset,
extents: Remove preserve_lru feature. preserve_lru feature adds lots of complication, for little value. Removing it means merged extents are re-added to the lru list, and may take longer to madvise away than they otherwise would. Canaries after removal seem flat for several services (no change).
2018-02-23 01:40:27 +08:00
extent_t *extent) {
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
assert(extent_arena_ind_get(extent) == arena_ind_get(arena));
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
assert(extent_state_get(extent) == eset_state_get(eset));
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Extent -> Eset: Move insertion and removal.
2019-09-21 14:51:13 +08:00
eset_remove_locked(tsdn, eset, extent);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_state_set(extent, extent_state_active);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static bool
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_rtree_leaf_elms_lookup(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx,
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
const extent_t *extent, bool dependent, bool init_missing,
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_elm_t **r_elm_a, rtree_leaf_elm_t **r_elm_b) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
*r_elm_a = rtree_leaf_elm_lookup(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_base_get(extent), dependent, init_missing);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!dependent && *r_elm_a == NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(*r_elm_a != NULL);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
*r_elm_b = rtree_leaf_elm_lookup(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)extent_last_get(extent), dependent, init_missing);
if (!dependent && *r_elm_b == NULL) {
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
assert(*r_elm_b != NULL);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static void
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
extent_rtree_write_acquired(tsdn_t *tsdn, rtree_leaf_elm_t *elm_a,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
rtree_leaf_elm_t *elm_b, extent_t *extent, szind_t szind, bool slab) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_leaf_elm_write(tsdn, &extents_rtree, elm_a, extent, szind, slab);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (elm_b != NULL) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_leaf_elm_write(tsdn, &extents_rtree, elm_b, extent, szind,
slab);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static void
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_interior_register(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx, extent_t *extent,
szind_t szind) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(extent_slab_get(extent));
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
/* Register interior. */
for (size_t i = 1; i < (extent_size_get(extent) >> LG_PAGE) - 1; i++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_write(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_base_get(extent) + (uintptr_t)(i <<
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
LG_PAGE), extent, szind, true);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
static void
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_gdump_add(tsdn_t *tsdn, const extent_t *extent) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
cassert(config_prof);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/* prof_gdump() requirement. */
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
if (opt_prof && extent_state_get(extent) == extent_state_active) {
Fix gdump triggering regression. Now that extents are not multiples of chunksize, it's necessary to track pages rather than chunks.
2016-06-06 05:43:20 +08:00
size_t nadd = extent_size_get(extent) >> LG_PAGE;
Convert extent module to use C11-style atomcis
2017-04-05 09:20:42 +08:00
size_t cur = atomic_fetch_add_zu(&curpages, nadd,
ATOMIC_RELAXED) + nadd;
size_t high = atomic_load_zu(&highpages, ATOMIC_RELAXED);
while (cur > high && !atomic_compare_exchange_weak_zu(
&highpages, &high, cur, ATOMIC_RELAXED, ATOMIC_RELAXED)) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/*
* Don't refresh cur, because it may have decreased
Fix gdump triggering regression. Now that extents are not multiples of chunksize, it's necessary to track pages rather than chunks.
2016-06-06 05:43:20 +08:00
* since this thread lost the highpages update race.
Convert extent module to use C11-style atomcis
2017-04-05 09:20:42 +08:00
* Note that high is updated in case of CAS failure.
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
*/
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (cur > high && prof_gdump_get_unlocked()) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
prof_gdump(tsdn);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
static void
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_gdump_sub(tsdn_t *tsdn, const extent_t *extent) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
cassert(config_prof);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
if (opt_prof && extent_state_get(extent) == extent_state_active) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
size_t nsub = extent_size_get(extent) >> LG_PAGE;
Convert extent module to use C11-style atomcis
2017-04-05 09:20:42 +08:00
assert(atomic_load_zu(&curpages, ATOMIC_RELAXED) >= nsub);
atomic_fetch_sub_zu(&curpages, nsub, ATOMIC_RELAXED);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
}
static bool
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_register_impl(tsdn_t *tsdn, extent_t *extent, bool gdump_add) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_elm_t *elm_a, *elm_b;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
/*
* We need to hold the lock to protect against a concurrent coalesce
* operation that sees us in a partial state.
*/
extent_lock(tsdn, extent);
if (extent_rtree_leaf_elms_lookup(tsdn, rtree_ctx, extent, false, true,
&elm_a, &elm_b)) {
Add the missing unlock in the error path of extent_register.
2019-03-30 04:27:20 +08:00
extent_unlock(tsdn, extent);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
szind_t szind = extent_szind_get_maybe_invalid(extent);
bool slab = extent_slab_get(extent);
extent_rtree_write_acquired(tsdn, elm_a, elm_b, extent, szind, slab);
if (slab) {
extent_interior_register(tsdn, rtree_ctx, extent, szind);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_unlock(tsdn, extent);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
Fix a race in extent_grow_retained(). Set extent as active prior to registration so that other threads can't modify it in the absence of locking. This regression was introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.), via non-obvious means. Removal of extents_mtx protection during extent_grow_retained() execution opened up the race, but in the presence of that locking, the code was safe. This resolves #599.
2017-02-04 16:43:32 +08:00
if (config_prof && gdump_add) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_gdump_add(tsdn, extent);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix a race in extent_grow_retained(). Set extent as active prior to registration so that other threads can't modify it in the absence of locking. This regression was introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.), via non-obvious means. Removal of extents_mtx protection during extent_grow_retained() execution opened up the race, but in the presence of that locking, the code was safe. This resolves #599.
2017-02-04 16:43:32 +08:00
static bool
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_register(tsdn_t *tsdn, extent_t *extent) {
Fix a race in extent_grow_retained(). Set extent as active prior to registration so that other threads can't modify it in the absence of locking. This regression was introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.), via non-obvious means. Removal of extents_mtx protection during extent_grow_retained() execution opened up the race, but in the presence of that locking, the code was safe. This resolves #599.
2017-02-04 16:43:32 +08:00
return extent_register_impl(tsdn, extent, true);
}
static bool
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_register_no_gdump_add(tsdn_t *tsdn, extent_t *extent) {
Fix a race in extent_grow_retained(). Set extent as active prior to registration so that other threads can't modify it in the absence of locking. This regression was introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.), via non-obvious means. Removal of extents_mtx protection during extent_grow_retained() execution opened up the race, but in the presence of that locking, the code was safe. This resolves #599.
2017-02-04 16:43:32 +08:00
return extent_register_impl(tsdn, extent, false);
}
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
static void
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_reregister(tsdn_t *tsdn, extent_t *extent) {
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
bool err = extent_register(tsdn, extent);
assert(!err);
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
* Removes all pointers to the given extent from the global rtree indices for
* its interior. This is relevant for slab extents, for which we need to do
* metadata lookups at places other than the head of the extent. We deregister
* on the interior, then, when an extent moves from being an active slab to an
* inactive state.
*/
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_deregister(tsdn_t *tsdn, rtree_ctx_t *rtree_ctx,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_t *extent) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t i;
assert(extent_slab_get(extent));
for (i = 1; i < (extent_size_get(extent) >> LG_PAGE) - 1; i++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_clear(tsdn, &extents_rtree, rtree_ctx,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
(uintptr_t)extent_base_get(extent) + (uintptr_t)(i <<
LG_PAGE));
}
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
* Removes all pointers to the given extent from the global rtree.
*/
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void
Fix extent deregister on the leak path. On leak path we should not adjust gdump when deregister.
2017-12-09 07:06:08 +08:00
extent_deregister_impl(tsdn_t *tsdn, extent_t *extent, bool gdump) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_elm_t *elm_a, *elm_b;
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_rtree_leaf_elms_lookup(tsdn, rtree_ctx, extent, true, false,
&elm_a, &elm_b);
extent_lock(tsdn, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_rtree_write_acquired(tsdn, elm_a, elm_b, NULL, SC_NSIZES, false);
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
if (extent_slab_get(extent)) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
extent_interior_deregister(tsdn, rtree_ctx, extent);
Fix an extent [de]allocation/[de]registration race. Deregister extents before deallocation, so that subsequent reallocation/registration doesn't race with deregistration.
2016-06-06 11:39:25 +08:00
extent_slab_set(extent, false);
}
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_unlock(tsdn, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Fix extent deregister on the leak path. On leak path we should not adjust gdump when deregister.
2017-12-09 07:06:08 +08:00
if (config_prof && gdump) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_gdump_sub(tsdn, extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
}
Fix extent deregister on the leak path. On leak path we should not adjust gdump when deregister.
2017-12-09 07:06:08 +08:00
static void
extent_deregister(tsdn_t *tsdn, extent_t *extent) {
extent_deregister_impl(tsdn, extent, true);
}
static void
extent_deregister_no_gdump_sub(tsdn_t *tsdn, extent_t *extent) {
extent_deregister_impl(tsdn, extent, false);
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* Tries to find and remove an extent from eset that can be used for the
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
* given allocation request.
*/
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static extent_t *
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_recycle_extract(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
void *new_addr, size_t size, size_t pad, size_t alignment, bool slab,
Avoid setting zero and commit if split fails in extent_recycle.
2017-11-17 06:27:23 +08:00
bool growing_retained) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Fix/simplify extent_recycle() allocation size computations. Do not call s2u() during alloc_size computation, since any necessary ceiling increase is taken care of later by extent_first_best_fit() --> extent_size_quantize_ceil(), and the s2u() call may erroneously cause a higher quantization result. Remove an overly strict overflow check that was added in 4a7852137d8b6598fdb90ea8e1fd3bc8a8b94a3a (Fix extent_recycle()'s cache-oblivious padding support.).
2016-11-04 14:49:21 +08:00
assert(alignment > 0);
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
if (config_debug && new_addr != NULL) {
/*
* Non-NULL new_addr has two use cases:
*
* 1) Recycle a known-extant extent, e.g. during purging.
* 2) Perform in-place expanding reallocation.
*
* Regardless of use case, new_addr must either refer to a
* non-existing extent, or to the base of an extant extent,
* since only active slabs support interior lookups (which of
* course cannot be recycled).
*/
assert(PAGE_ADDR2BASE(new_addr) == new_addr);
Fix size class overflow bugs. Avoid calling s2u() on raw extent sizes in extent_recycle(). Clamp psz2ind() (implemented as psz2ind_clamp()) when inserting/removing into/from size-segregated extent heaps.
2016-10-04 05:18:55 +08:00
assert(pad == 0);
assert(alignment <= PAGE);
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t esize = size + pad;
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_lock(tsdn, &eset->mtx);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_t *extent;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (new_addr != NULL) {
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent = extent_lock_from_addr(tsdn, rtree_ctx, new_addr,
false);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
if (extent != NULL) {
/*
* We might null-out extent to report an error, but we
* still need to unlock the associated mutex after.
*/
extent_t *unlock_extent = extent;
assert(extent_base_get(extent) == new_addr);
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
if (extent_arena_ind_get(extent)
!= arena_ind_get(arena) ||
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_size_get(extent) < esize ||
extent_state_get(extent) !=
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
eset_state_get(eset)) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent = NULL;
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
}
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_unlock(tsdn, unlock_extent);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add various mutex ownership assertions.
2016-09-24 03:18:36 +08:00
} else {
Extent -> Eset: Move extent fit functions.
2019-09-22 00:36:22 +08:00
extent = eset_fit_locked(tsdn, eset, esize, alignment);
Add various mutex ownership assertions.
2016-09-24 03:18:36 +08:00
}
Add new_addr validation in extent_recycle().
2016-09-24 03:11:01 +08:00
if (extent == NULL) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_unlock(tsdn, &eset->mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_activate_locked(tsdn, arena, eset, extent);
malloc_mutex_unlock(tsdn, &eset->mtx);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
return extent;
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
* Given an allocation request and an extent guaranteed to be able to satisfy
* it, this splits off lead and trail extents, leaving extent pointing to an
* extent satisfying the allocation.
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* This function doesn't put lead or trail into any eset_t; it's the caller's
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
* job to ensure that they can be reused.
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
*/
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
typedef enum {
/*
* Split successfully. lead, extent, and trail, are modified to extents
* describing the ranges before, in, and after the given allocation.
*/
extent_split_interior_ok,
/*
* The extent can't satisfy the given allocation request. None of the
* input extent_t *s are touched.
*/
extent_split_interior_cant_alloc,
/*
* In a potentially invalid state. Must leak (if *to_leak is non-NULL),
* and salvage what's still salvageable (if *to_salvage is non-NULL).
* None of lead, extent, or trail are valid.
*/
extent_split_interior_error
} extent_split_interior_result_t;
static extent_split_interior_result_t
extent_split_interior(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
/* The result of splitting, in case of success. */
extent_t **extent, extent_t **lead, extent_t **trail,
/* The mess to clean up, in case of error. */
extent_t **to_leak, extent_t **to_salvage,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
void *new_addr, size_t size, size_t pad, size_t alignment, bool slab,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
szind_t szind, bool growing_retained) {
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t esize = size + pad;
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
size_t leadsize = ALIGNMENT_CEILING((uintptr_t)extent_base_get(*extent),
PAGE_CEILING(alignment)) - (uintptr_t)extent_base_get(*extent);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
assert(new_addr == NULL || leadsize == 0);
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
if (extent_size_get(*extent) < leadsize + esize) {
return extent_split_interior_cant_alloc;
}
size_t trailsize = extent_size_get(*extent) - leadsize - esize;
*lead = NULL;
*trail = NULL;
*to_leak = NULL;
*to_salvage = NULL;
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* Split the lead. */
if (leadsize != 0) {
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
*lead = *extent;
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
*extent = extent_split_impl(tsdn, arena, extent_hooks,
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
*lead, leadsize, SC_NSIZES, false, esize + trailsize, szind,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
slab, growing_retained);
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
if (*extent == NULL) {
*to_leak = *lead;
*lead = NULL;
return extent_split_interior_error;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
}
/* Split the trail. */
if (trailsize != 0) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
*trail = extent_split_impl(tsdn, arena, extent_hooks, *extent,
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
esize, szind, slab, trailsize, SC_NSIZES, false,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
growing_retained);
if (*trail == NULL) {
*to_leak = *extent;
*to_salvage = *lead;
*lead = NULL;
*extent = NULL;
return extent_split_interior_error;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
}
if (leadsize == 0 && trailsize == 0) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/*
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
* Splitting causes szind to be set as a side effect, but no
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
* splitting occurred.
*/
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
extent_szind_set(*extent, szind);
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
if (szind != SC_NSIZES) {
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
rtree_szind_slab_update(tsdn, &extents_rtree, rtree_ctx,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
(uintptr_t)extent_addr_get(*extent), szind, slab);
if (slab && extent_size_get(*extent) > PAGE) {
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
rtree_szind_slab_update(tsdn, &extents_rtree,
rtree_ctx,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
(uintptr_t)extent_past_get(*extent) -
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
(uintptr_t)PAGE, szind, slab);
}
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
return extent_split_interior_ok;
}
/*
* This fulfills the indicated allocation request out of the given extent (which
* the caller should have ensured was big enough). If there's any unused space
* before or after the resulting allocation, that space is given its own extent
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* and put back into eset.
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
*/
static extent_t *
extent_recycle_split(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
void *new_addr, size_t size, size_t pad, size_t alignment, bool slab,
szind_t szind, extent_t *extent, bool growing_retained) {
extent_t *lead;
extent_t *trail;
extent_t *to_leak;
extent_t *to_salvage;
extent_split_interior_result_t result = extent_split_interior(
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
tsdn, arena, extent_hooks, rtree_ctx, &extent, &lead, &trail,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
&to_leak, &to_salvage, new_addr, size, pad, alignment, slab, szind,
growing_retained);
Avoid leaking extents / VM when split is not supported. This can only happen on Windows and with opt.retain disabled (which isn't the default). The solution is suboptimal, however not a common case as retain is the long term plan for all platforms anyway.
2019-07-16 03:16:02 +08:00
if (!maps_coalesce && result != extent_split_interior_ok
&& !opt_retain) {
/*
* Split isn't supported (implies Windows w/o retain). Avoid
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* leaking the eset.
Avoid leaking extents / VM when split is not supported. This can only happen on Windows and with opt.retain disabled (which isn't the default). The solution is suboptimal, however not a common case as retain is the long term plan for all platforms anyway.
2019-07-16 03:16:02 +08:00
*/
assert(to_leak != NULL && lead == NULL && trail == NULL);
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate(tsdn, arena, eset, to_leak);
Avoid leaking extents / VM when split is not supported. This can only happen on Windows and with opt.retain disabled (which isn't the default). The solution is suboptimal, however not a common case as retain is the long term plan for all platforms anyway.
2019-07-16 03:16:02 +08:00
return NULL;
}
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
if (result == extent_split_interior_ok) {
if (lead != NULL) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate(tsdn, arena, eset, lead);
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
}
if (trail != NULL) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate(tsdn, arena, eset, trail);
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
}
return extent;
} else {
/*
* We should have picked an extent that was large enough to
* fulfill our allocation request.
*/
assert(result == extent_split_interior_error);
if (to_salvage != NULL) {
extent_deregister(tsdn, to_salvage);
}
if (to_leak != NULL) {
Add more tests for extent hooks failure paths.
2017-11-29 04:21:58 +08:00
void *leak = extent_base_get(to_leak);
Fix extent deregister on the leak path. On leak path we should not adjust gdump when deregister.
2017-12-09 07:06:08 +08:00
extent_deregister_no_gdump_sub(tsdn, to_leak);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extents_abandon_vm(tsdn, arena, extent_hooks, eset,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
to_leak, growing_retained);
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
assert(extent_lock_from_addr(tsdn, rtree_ctx, leak,
false) == NULL);
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
}
return NULL;
}
unreachable();
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
Avoid potential issues on extent zero-out. When custom extent_hooks or transparent huge pages are in use, the purging semantics may change, which means we may not get zeroed pages on repopulating. Fixing the issue by manually memset for such cases.
2018-08-11 07:08:50 +08:00
static bool
extent_need_manual_zero(arena_t *arena) {
/*
* Need to manually zero the extent on repopulating if either; 1) non
* default extent hooks installed (in which case the purge semantics may
* change); or 2) transparent huge pages enabled.
*/
return (!arena_has_default_hooks(arena) ||
(opt_thp == thp_mode_always));
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
* Tries to satisfy the given allocation request by reusing one of the extents
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* in the given eset_t.
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
*/
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
static extent_t *
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_recycle(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, void *new_addr, size_t size, size_t pad,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t alignment, bool slab, szind_t szind, bool *zero, bool *commit,
bool growing_retained) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
assert(new_addr == NULL || !slab);
assert(pad == 0 || !slab);
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
assert(!*zero || !slab);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_t *extent = extent_recycle_extract(tsdn, arena, extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
rtree_ctx, eset, new_addr, size, pad, alignment, slab,
Avoid setting zero and commit if split fails in extent_recycle.
2017-11-17 06:27:23 +08:00
growing_retained);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
if (extent == NULL) {
return NULL;
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent = extent_recycle_split(tsdn, arena, extent_hooks, rtree_ctx,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset, new_addr, size, pad, alignment, slab, szind, extent,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
growing_retained);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
if (extent == NULL) {
return NULL;
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
if (*commit && !extent_committed_get(extent)) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_commit_impl(tsdn, arena, extent_hooks, extent,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
0, extent_size_get(extent), growing_retained)) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks, eset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
extent, growing_retained);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Avoid potential issues on extent zero-out. When custom extent_hooks or transparent huge pages are in use, the purging semantics may change, which means we may not get zeroed pages on repopulating. Fixing the issue by manually memset for such cases.
2018-08-11 07:08:50 +08:00
if (!extent_need_manual_zero(arena)) {
extent_zeroed_set(extent, true);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Avoid setting zero and commit if split fails in extent_recycle.
2017-11-17 06:27:23 +08:00
if (extent_committed_get(extent)) {
*commit = true;
}
if (extent_zeroed_get(extent)) {
*zero = true;
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (pad != 0) {
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_addr_randomize(tsdn, arena, extent, alignment);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
assert(extent_state_get(extent) == extent_state_active);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (slab) {
extent_slab_set(extent, slab);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_interior_register(tsdn, rtree_ctx, extent, szind);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
if (*zero) {
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
void *addr = extent_base_get(extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (!extent_zeroed_get(extent)) {
Avoid touching all pages in extent_recycle for debug build. We may have a large number of pages with *zero set (since they are populated on demand). Only check the first page to avoid paging in all of them.
2018-11-13 03:15:03 +08:00
size_t size = extent_size_get(extent);
Avoid potential issues on extent zero-out. When custom extent_hooks or transparent huge pages are in use, the purging semantics may change, which means we may not get zeroed pages on repopulating. Fixing the issue by manually memset for such cases.
2018-08-11 07:08:50 +08:00
if (extent_need_manual_zero(arena) ||
pages_purge_forced(addr, size)) {
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
memset(addr, 0, size);
Prefer pages_purge_forced() over memset(). This has the dual advantages of allowing for sparsely used large allocations, and relying on the kernel to supply zeroed pages, which tends to be very fast on modern systems.
2017-03-10 09:20:00 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
} else if (config_debug) {
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t *p = (size_t *)(uintptr_t)addr;
Avoid touching all pages in extent_recycle for debug build. We may have a large number of pages with *zero set (since they are populated on demand). Only check the first page to avoid paging in all of them.
2018-11-13 03:15:03 +08:00
/* Check the first page only. */
for (size_t i = 0; i < PAGE / sizeof(size_t); i++) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(p[i] == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
/*
* If the caller specifies (!*zero), it is still possible to receive zeroed
* memory, in which case *zero is toggled to true. arena_extent_alloc() takes
* advantage of this to avoid demanding zeroed extents, but taking advantage of
* them if they are returned.
*/
static void *
extent_alloc_core(tsdn_t *tsdn, arena_t *arena, void *new_addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t alignment, bool *zero, bool *commit, dss_prec_t dss_prec) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
void *ret;
assert(size != 0);
assert(alignment != 0);
/* "primary" dss. */
if (have_dss && dss_prec == dss_prec_primary && (ret =
extent_alloc_dss(tsdn, arena, new_addr, size, alignment, zero,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
commit)) != NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* mmap. */
if ((ret = extent_alloc_mmap(new_addr, size, alignment, zero, commit))
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
!= NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* "secondary" dss. */
if (have_dss && dss_prec == dss_prec_secondary && (ret =
extent_alloc_dss(tsdn, arena, new_addr, size, alignment, zero,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
commit)) != NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* All strategies for allocation failed. */
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
static void *
extent_alloc_default_impl(tsdn_t *tsdn, arena_t *arena, void *new_addr,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t size, size_t alignment, bool *zero, bool *commit) {
Add opt.thp which allows explicit hugepage usage. "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
2018-02-17 06:19:19 +08:00
void *ret = extent_alloc_core(tsdn, arena, new_addr, size, alignment, zero,
Transition arena struct fields to C11 atomics
2017-04-05 08:22:24 +08:00
commit, (dss_prec_t)atomic_load_u(&arena->dss_prec,
ATOMIC_RELAXED));
Add opt.thp which allows explicit hugepage usage. "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
2018-02-17 06:19:19 +08:00
if (have_madvise_huge && ret) {
pages_set_thp_state(ret, size);
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static void *
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_alloc_default(extent_hooks_t *extent_hooks, void *new_addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t alignment, bool *zero, bool *commit, unsigned arena_ind) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
tsdn_t *tsdn;
arena_t *arena;
tsdn = tsdn_fetch();
arena = arena_get(tsdn, arena_ind, false);
/*
* The arena we're allocating on behalf of must have been initialized
* already.
*/
assert(arena != NULL);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent_alloc_default_impl(tsdn, arena, new_addr, size,
Ensure page alignment on extent_alloc. This is discovered and suggested by @jasone in #1468. When custom extent hooks are in use, we should ensure page alignment on the extent alloc path, instead of relying on the user hooks to do so.
2019-03-29 11:42:40 +08:00
ALIGNMENT_CEILING(alignment, PAGE), zero, commit);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
static void
extent_hook_pre_reentrancy(tsdn_t *tsdn, arena_t *arena) {
tsd_t *tsd = tsdn_null(tsdn) ? tsd_fetch() : tsdn_tsd(tsdn);
Relax constraints on reentrancy for extent hooks. If we guarantee no malloc activity in extent hooks, it's possible to make customized hooks working on arena 0. Remove the non-a0 assertion to enable such use cases.
2017-08-31 07:17:04 +08:00
if (arena == arena_get(tsd_tsdn(tsd), 0, false)) {
/*
* The only legitimate case of customized extent hooks for a0 is
* hooks with no allocation activities. One such example is to
* place metadata on pre-allocated resources such as huge pages.
* In that case, rely on reentrancy_level checks to catch
* infinite recursions.
*/
pre_reentrancy(tsd, NULL);
} else {
pre_reentrancy(tsd, arena);
}
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
}
static void
extent_hook_post_reentrancy(tsdn_t *tsdn) {
tsd_t *tsd = tsdn_null(tsdn) ? tsd_fetch() : tsdn_tsd(tsdn);
post_reentrancy(tsd);
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
/*
* If virtual memory is retained, create increasingly larger extents from which
* to split requested extents in order to limit the total number of disjoint
* virtual memory ranges retained by each arena.
*/
static extent_t *
extent_grow_retained(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, size_t size, size_t pad, size_t alignment,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool slab, szind_t szind, bool *zero, bool *commit) {
malloc_mutex_assert_owner(tsdn, &arena->extent_grow_mtx);
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
assert(pad == 0 || !slab);
assert(!*zero || !slab);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t esize = size + pad;
size_t alloc_size_min = esize + PAGE_CEILING(alignment) - PAGE;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
/* Beware size_t wrap-around. */
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
if (alloc_size_min < esize) {
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
goto label_err;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
/*
* Find the next extent size in the series that would be large enough to
* satisfy this request.
*/
pszind_t egn_skip = 0;
Header refactoring: Pull size helpers out of jemalloc module.
2017-05-31 01:45:37 +08:00
size_t alloc_size = sz_pind2sz(arena->extent_grow_next + egn_skip);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
while (alloc_size < alloc_size_min) {
egn_skip++;
SC: Remove global data. The global data is mostly only used at initialization, or for easy access to values we could compute statically. Instead of consuming that space (and risking TLB misses), we can just pass around a pointer to stack data during bootstrapping.
2018-07-20 08:08:10 +08:00
if (arena->extent_grow_next + egn_skip >=
sz_psz2ind(SC_LARGE_MAXCLASS)) {
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
/* Outside legal range. */
goto label_err;
}
Header refactoring: Pull size helpers out of jemalloc module.
2017-05-31 01:45:37 +08:00
alloc_size = sz_pind2sz(arena->extent_grow_next + egn_skip);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
extent_t *extent = extent_alloc(tsdn, arena);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (extent == NULL) {
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
goto label_err;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
bool zeroed = false;
bool committed = false;
Fix extent_hooks in extent_grow_retained(). This issue caused the default extent alloc function to be incorrectly used even when arena.<i>.extent_hooks is set. This bug was introduced by 411697adcda2fd75e135cdcdafb95f2bd295dc7f (Use exponential series to size extents.), which was first released in 5.0.0.
2017-06-14 07:16:33 +08:00
void *ptr;
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks == &extent_hooks_default) {
Add opt.thp which allows explicit hugepage usage. "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
2018-02-17 06:19:19 +08:00
ptr = extent_alloc_default_impl(tsdn, arena, NULL,
alloc_size, PAGE, &zeroed, &committed);
Fix extent_hooks in extent_grow_retained(). This issue caused the default extent alloc function to be incorrectly used even when arena.<i>.extent_hooks is set. This bug was introduced by 411697adcda2fd75e135cdcdafb95f2bd295dc7f (Use exponential series to size extents.), which was first released in 5.0.0.
2017-06-14 07:16:33 +08:00
} else {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
ptr = extent_hooks->alloc(extent_hooks, NULL,
Fix extent_hooks in extent_grow_retained(). This issue caused the default extent alloc function to be incorrectly used even when arena.<i>.extent_hooks is set. This bug was introduced by 411697adcda2fd75e135cdcdafb95f2bd295dc7f (Use exponential series to size extents.), which was first released in 5.0.0.
2017-06-14 07:16:33 +08:00
alloc_size, PAGE, &zeroed, &committed,
arena_ind_get(arena));
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Fix extent_hooks in extent_grow_retained(). This issue caused the default extent alloc function to be incorrectly used even when arena.<i>.extent_hooks is set. This bug was introduced by 411697adcda2fd75e135cdcdafb95f2bd295dc7f (Use exponential series to size extents.), which was first released in 5.0.0.
2017-06-14 07:16:33 +08:00
}
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_init(extent, arena_ind_get(arena), ptr, alloc_size, false,
SC_NSIZES, arena_extent_sn_next(arena), extent_state_active, zeroed,
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
committed, true, EXTENT_IS_HEAD);
Fix OOM paths in extent_grow_retained().
2017-05-26 09:15:41 +08:00
if (ptr == NULL) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
extent_dalloc(tsdn, arena, extent);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
goto label_err;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Add a "dumpable" bit to the extent state. Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
2017-09-19 08:25:57 +08:00
Fix OOM paths in extent_grow_retained().
2017-05-26 09:15:41 +08:00
if (extent_register_no_gdump_add(tsdn, extent)) {
extent_dalloc instead of leak when register fails. extent_register may only fail if the underlying extent and region got stolen / coalesced before we lock. Avoid doing extent_leak (which purges the region) since we don't really own the region.
2019-07-16 05:55:43 +08:00
extent_dalloc(tsdn, arena, extent);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
goto label_err;
Fix OOM paths in extent_grow_retained().
2017-05-26 09:15:41 +08:00
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
Trim before commit in extent_recycle(). This avoids creating clean committed pages as a side effect of aligned allocation. For configurations that decommit memory, purged pages are decommitted, and decommitted extents cannot be coalesced with committed extents. Unless the clean committed pages happen to be selected during allocation, they cause unnecessary permanent extent fragmentation. This resolves #766.
2017-04-20 07:14:54 +08:00
if (extent_zeroed_get(extent) && extent_committed_get(extent)) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
*zero = true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
if (extent_committed_get(extent)) {
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
*commit = true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
extent_t *lead;
extent_t *trail;
extent_t *to_leak;
extent_t *to_salvage;
extent_split_interior_result_t result = extent_split_interior(
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
tsdn, arena, extent_hooks, rtree_ctx, &extent, &lead, &trail,
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
&to_leak, &to_salvage, NULL, size, pad, alignment, slab, szind,
true);
if (result == extent_split_interior_ok) {
if (lead != NULL) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks,
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
&arena->eset_retained, lead, true);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
if (trail != NULL) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks,
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
&arena->eset_retained, trail, true);
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
}
} else {
/*
* We should have allocated a sufficiently large extent; the
* cant_alloc case should not occur.
*/
assert(result == extent_split_interior_error);
Avoid a resource leak down extent split failure paths. Previously, we would leak the extent and memory associated with a salvageable portion of an extent that we were trying to split in three, in the case where the first split attempt succeeded and the second failed.
2018-04-18 04:16:42 +08:00
if (to_salvage != NULL) {
if (config_prof) {
extent_gdump_add(tsdn, to_salvage);
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks,
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
&arena->eset_retained, to_salvage, true);
Avoid a resource leak down extent split failure paths. Previously, we would leak the extent and memory associated with a salvageable portion of an extent that we were trying to split in three, in the case where the first split attempt succeeded and the second failed.
2018-04-18 04:16:42 +08:00
}
Factor out extent-splitting core from extent lifetime management. Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
2017-08-30 07:50:57 +08:00
if (to_leak != NULL) {
Fix extent deregister on the leak path. On leak path we should not adjust gdump when deregister.
2017-12-09 07:06:08 +08:00
extent_deregister_no_gdump_sub(tsdn, to_leak);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extents_abandon_vm(tsdn, arena, extent_hooks,
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
&arena->eset_retained, to_leak, true);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Avoid a resource leak down extent split failure paths. Previously, we would leak the extent and memory associated with a salvageable portion of an extent that we were trying to split in three, in the case where the first split attempt succeeded and the second failed.
2018-04-18 04:16:42 +08:00
goto label_err;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
if (*commit && !extent_committed_get(extent)) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_commit_impl(tsdn, arena, extent_hooks, extent, 0,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
extent_size_get(extent), true)) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks,
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
&arena->eset_retained, extent, true);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
goto label_err;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Avoid potential issues on extent zero-out. When custom extent_hooks or transparent huge pages are in use, the purging semantics may change, which means we may not get zeroed pages on repopulating. Fixing the issue by manually memset for such cases.
2018-08-11 07:08:50 +08:00
if (!extent_need_manual_zero(arena)) {
extent_zeroed_set(extent, true);
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
/*
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
* Increment extent_grow_next if doing so wouldn't exceed the allowed
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
* range.
*/
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
if (arena->extent_grow_next + egn_skip + 1 <=
arena->retain_grow_limit) {
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
arena->extent_grow_next += egn_skip + 1;
} else {
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
arena->extent_grow_next = arena->retain_grow_limit;
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
}
/* All opportunities for failure are past. */
malloc_mutex_unlock(tsdn, &arena->extent_grow_mtx);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
if (config_prof) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/* Adjust gdump stats now that extent is final size. */
extent_gdump_add(tsdn, extent);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (pad != 0) {
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_addr_randomize(tsdn, arena, extent, alignment);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
if (slab) {
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn,
&rtree_ctx_fallback);
extent_slab_set(extent, true);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_interior_register(tsdn, rtree_ctx, extent, szind);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (*zero && !extent_zeroed_get(extent)) {
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
void *addr = extent_base_get(extent);
size_t size = extent_size_get(extent);
Avoid potential issues on extent zero-out. When custom extent_hooks or transparent huge pages are in use, the purging semantics may change, which means we may not get zeroed pages on repopulating. Fixing the issue by manually memset for such cases.
2018-08-11 07:08:50 +08:00
if (extent_need_manual_zero(arena) ||
pages_purge_forced(addr, size)) {
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
memset(addr, 0, size);
Prefer pages_purge_forced() over memset(). This has the dual advantages of allowing for sparsely used large allocations, and relying on the kernel to supply zeroed pages, which tends to be very fast on modern systems.
2017-03-10 09:20:00 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent;
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
label_err:
malloc_mutex_unlock(tsdn, &arena->extent_grow_mtx);
return NULL;
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static extent_t *
extent_alloc_retained(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, void *new_addr, size_t size, size_t pad,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t alignment, bool slab, szind_t szind, bool *zero, bool *commit) {
assert(size != 0);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(alignment != 0);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
malloc_mutex_lock(tsdn, &arena->extent_grow_mtx);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_t *extent = extent_recycle(tsdn, arena, extent_hooks,
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
&arena->eset_retained, new_addr, size, pad, alignment, slab,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
szind, zero, commit, true);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if (extent != NULL) {
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
malloc_mutex_unlock(tsdn, &arena->extent_grow_mtx);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (config_prof) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_gdump_add(tsdn, extent);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
} else if (opt_retain && new_addr == NULL) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent = extent_grow_retained(tsdn, arena, extent_hooks, size,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
pad, alignment, slab, szind, zero, commit);
/* extent_grow_retained() always releases extent_grow_mtx. */
} else {
malloc_mutex_unlock(tsdn, &arena->extent_grow_mtx);
Use exponential series to size extents. If virtual memory is retained, allocate extents such that their sizes form an exponentially growing series. This limits the number of disjoint virtual memory ranges so that extent merging can be effective even if multiple arenas' extent allocation requests are highly interleaved. This resolves #462.
2016-11-22 15:23:03 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
malloc_mutex_assert_not_owner(tsdn, &arena->extent_grow_mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static extent_t *
extent_alloc_wrapper_hard(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, void *new_addr, size_t size, size_t pad,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t alignment, bool slab, szind_t szind, bool *zero, bool *commit) {
size_t esize = size + pad;
extent_t *extent = extent_alloc(tsdn, arena);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (extent == NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
void *addr;
Ensure page alignment on extent_alloc. This is discovered and suggested by @jasone in #1468. When custom extent hooks are in use, we should ensure page alignment on the extent alloc path, instead of relying on the user hooks to do so.
2019-03-29 11:42:40 +08:00
size_t palignment = ALIGNMENT_CEILING(alignment, PAGE);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks == &extent_hooks_default) {
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
/* Call directly to propagate tsdn. */
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
addr = extent_alloc_default_impl(tsdn, arena, new_addr, esize,
Ensure page alignment on extent_alloc. This is discovered and suggested by @jasone in #1468. When custom extent hooks are in use, we should ensure page alignment on the extent alloc path, instead of relying on the user hooks to do so.
2019-03-29 11:42:40 +08:00
palignment, zero, commit);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
} else {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
addr = extent_hooks->alloc(extent_hooks, new_addr,
Ensure page alignment on extent_alloc. This is discovered and suggested by @jasone in #1468. When custom extent hooks are in use, we should ensure page alignment on the extent alloc path, instead of relying on the user hooks to do so.
2019-03-29 11:42:40 +08:00
esize, palignment, zero, commit, arena_ind_get(arena));
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (addr == NULL) {
extent_dalloc(tsdn, arena, extent);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_init(extent, arena_ind_get(arena), addr, esize, slab, szind,
Fix arguments passed to extent_init.
2018-04-07 04:18:21 +08:00
arena_extent_sn_next(arena), extent_state_active, *zero, *commit,
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
true, EXTENT_NOT_HEAD);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (pad != 0) {
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_addr_randomize(tsdn, arena, extent, alignment);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (extent_register(tsdn, extent)) {
extent_dalloc instead of leak when register fails. extent_register may only fail if the underlying extent and region got stolen / coalesced before we lock. Avoid doing extent_leak (which purges the region) since we don't really own the region.
2019-07-16 05:55:43 +08:00
extent_dalloc(tsdn, arena, extent);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
extent_t *
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_alloc_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, void *new_addr, size_t size, size_t pad,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
size_t alignment, bool slab, szind_t szind, bool *zero, bool *commit) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_t *extent = extent_alloc_retained(tsdn, arena, extent_hooks,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
new_addr, size, pad, alignment, slab, szind, zero, commit);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (extent == NULL) {
Bypass extent_alloc_wrapper_hard for no_move_expand. When retain is enabled, we should not attempt mmap for in-place expansion (large_ralloc_no_move), because it's virtually impossible to succeed, and causes unnecessary syscalls (which can cause lock contention under load).
2017-08-01 04:01:07 +08:00
if (opt_retain && new_addr != NULL) {
/*
* When retain is enabled and new_addr is set, we do not
* attempt extent_alloc_wrapper_hard which does mmap
* that is very unlikely to succeed (unless it happens
* to be at the end).
*/
return NULL;
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent = extent_alloc_wrapper_hard(tsdn, arena, extent_hooks,
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
new_addr, size, pad, alignment, slab, szind, zero, commit);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Add a "dumpable" bit to the extent state. Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
2017-09-19 08:25:57 +08:00
assert(extent == NULL || extent_dumpable_get(extent));
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
static bool
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_can_coalesce(arena_t *arena, eset_t *eset, const extent_t *inner,
Optimize extent coalescing. Refactor extent_can_coalesce(), extent_coalesce(), and extent_record() to avoid needlessly repeating extent [de]activation operations.
2017-02-13 15:00:19 +08:00
const extent_t *outer) {
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
assert(extent_arena_ind_get(inner) == arena_ind_get(arena));
if (extent_arena_ind_get(outer) != arena_ind_get(arena)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Optimize extent coalescing. Refactor extent_can_coalesce(), extent_coalesce(), and extent_record() to avoid needlessly repeating extent [de]activation operations.
2017-02-13 15:00:19 +08:00
assert(extent_state_get(inner) == extent_state_active);
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
if (extent_state_get(outer) != eset->state) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Optimize extent coalescing. Refactor extent_can_coalesce(), extent_coalesce(), and extent_record() to avoid needlessly repeating extent [de]activation operations.
2017-02-13 15:00:19 +08:00
if (extent_committed_get(inner) != extent_committed_get(outer)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
static bool
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_coalesce(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, extent_t *inner, extent_t *outer, bool forward,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool growing_retained) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
assert(extent_can_coalesce(arena, eset, inner, outer));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_activate_locked(tsdn, arena, eset, outer);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_unlock(tsdn, &eset->mtx);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
bool err = extent_merge_impl(tsdn, arena, extent_hooks,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
forward ? inner : outer, forward ? outer : inner, growing_retained);
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_lock(tsdn, &eset->mtx);
Optimize extent coalescing. Refactor extent_can_coalesce(), extent_coalesce(), and extent_record() to avoid needlessly repeating extent [de]activation operations.
2017-02-13 15:00:19 +08:00
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
if (err) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate_locked(tsdn, arena, eset, outer);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Optimize extent coalescing. Refactor extent_can_coalesce(), extent_coalesce(), and extent_record() to avoid needlessly repeating extent [de]activation operations.
2017-02-13 15:00:19 +08:00
return err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
static extent_t *
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_try_coalesce_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_t *extent, bool *coalesced, bool growing_retained,
bool inactive_only) {
/*
* We avoid checking / locking inactive neighbors for large size
* classes, since they are eagerly coalesced on deallocation which can
* cause lock contention.
*/
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/*
* Continue attempting to coalesce until failure, to protect against
* races with other threads that are thwarted by this one.
*/
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
bool again;
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
do {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
again = false;
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/* Try to coalesce forward. */
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_t *next = extent_lock_from_addr(tsdn, rtree_ctx,
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_past_get(extent), inactive_only);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
if (next != NULL) {
Fix extent_record(). Read adjacent rtree elements while holding element locks, since the extents mutex only protects against relevant like-state extent mutation. Fix management of the 'coalesced' loop state variable to merge forward/backward results, rather than overwriting the result of forward coalescing if attempting to coalesce backward. In practice this caused no correctness issues, but could cause extra iterations in rare cases. These regressions were introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.).
2017-02-06 15:59:53 +08:00
/*
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* eset->mtx only protects against races for
* like-state eset, so call extent_can_coalesce()
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
* before releasing next's pool lock.
Fix extent_record(). Read adjacent rtree elements while holding element locks, since the extents mutex only protects against relevant like-state extent mutation. Fix management of the 'coalesced' loop state variable to merge forward/backward results, rather than overwriting the result of forward coalescing if attempting to coalesce backward. In practice this caused no correctness issues, but could cause extra iterations in rare cases. These regressions were introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.).
2017-02-06 15:59:53 +08:00
*/
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
bool can_coalesce = extent_can_coalesce(arena, eset,
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent, next);
extent_unlock(tsdn, next);
Fix extent_record(). Read adjacent rtree elements while holding element locks, since the extents mutex only protects against relevant like-state extent mutation. Fix management of the 'coalesced' loop state variable to merge forward/backward results, rather than overwriting the result of forward coalescing if attempting to coalesce backward. In practice this caused no correctness issues, but could cause extra iterations in rare cases. These regressions were introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.).
2017-02-06 15:59:53 +08:00
if (can_coalesce && !extent_coalesce(tsdn, arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks, eset, extent, next, true,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
growing_retained)) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
if (eset->delay_coalesce) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
/* Do minimal coalescing. */
*coalesced = true;
return extent;
}
again = true;
Fix extent_record(). Read adjacent rtree elements while holding element locks, since the extents mutex only protects against relevant like-state extent mutation. Fix management of the 'coalesced' loop state variable to merge forward/backward results, rather than overwriting the result of forward coalescing if attempting to coalesce backward. In practice this caused no correctness issues, but could cause extra iterations in rare cases. These regressions were introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.).
2017-02-06 15:59:53 +08:00
}
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/* Try to coalesce backward. */
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_t *prev = extent_lock_from_addr(tsdn, rtree_ctx,
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_before_get(extent), inactive_only);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
if (prev != NULL) {
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
bool can_coalesce = extent_can_coalesce(arena, eset,
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent, prev);
extent_unlock(tsdn, prev);
Fix extent_record(). Read adjacent rtree elements while holding element locks, since the extents mutex only protects against relevant like-state extent mutation. Fix management of the 'coalesced' loop state variable to merge forward/backward results, rather than overwriting the result of forward coalescing if attempting to coalesce backward. In practice this caused no correctness issues, but could cause extra iterations in rare cases. These regressions were introduced by d27f29b468ae3e9d2b1da4a9880351d76e5a1662 (Disentangle arena and extent locking.).
2017-02-06 15:59:53 +08:00
if (can_coalesce && !extent_coalesce(tsdn, arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks, eset, extent, prev, false,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
growing_retained)) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent = prev;
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
if (eset->delay_coalesce) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
/* Do minimal coalescing. */
*coalesced = true;
return extent;
}
again = true;
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
}
}
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
} while (again);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
if (eset->delay_coalesce) {
Perform delayed coalescing prior to purging. Rather than purging uncoalesced extents, perform just enough incremental coalescing to purge only fully coalesced extents. In the absence of cached extent reuse, the immediate versus delayed incremental purging algorithms result in the same purge order. This resolves #655.
2017-03-03 10:04:35 +08:00
*coalesced = false;
}
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
return extent;
}
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
static extent_t *
extent_try_coalesce(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_t *extent, bool *coalesced, bool growing_retained) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
return extent_try_coalesce_impl(tsdn, arena, extent_hooks, rtree_ctx,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset, extent, coalesced, growing_retained, false);
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
}
static extent_t *
extent_try_coalesce_large(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, rtree_ctx_t *rtree_ctx, eset_t *eset,
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent_t *extent, bool *coalesced, bool growing_retained) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
return extent_try_coalesce_impl(tsdn, arena, extent_hooks, rtree_ctx,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset, extent, coalesced, growing_retained, true);
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
* Does the metadata management portions of putting an unused extent into the
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
* given eset_t (coalesces, deregisters slab interiors, the heap operations).
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
*/
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
static void
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn_t *tsdn, arena_t *arena, extent_hooks_t *extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
eset_t *eset, extent_t *extent, bool growing_retained) {
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Extents -> Eset: move extents_state_get.
2019-09-21 11:45:16 +08:00
assert((eset_state_get(eset) != extent_state_dirty &&
eset_state_get(eset) != extent_state_muzzy) ||
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
!extent_zeroed_get(extent));
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_lock(tsdn, &eset->mtx);
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_szind_set(extent, SC_NSIZES);
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
if (extent_slab_get(extent)) {
extent_interior_deregister(tsdn, rtree_ctx, extent);
extent_slab_set(extent, false);
}
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
assert(rtree_extent_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)extent_base_get(extent), true) == extent);
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
if (!eset->delay_coalesce) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent = extent_try_coalesce(tsdn, arena, extent_hooks,
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
rtree_ctx, eset, extent, NULL, growing_retained);
SC: Make some key size classes static. The largest small class, smallest large class, and largest large class may all be needed down fast paths; to avoid the risk of touching another cache line, we can make them available as constants.
2018-07-12 07:05:58 +08:00
} else if (extent_size_get(extent) >= SC_LARGE_MINCLASS) {
Extent -> Eset: Rename arena members.
2019-09-22 01:35:47 +08:00
assert(eset == &arena->eset_dirty);
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
/* Always coalesce large eset eagerly. */
Eagerly coalesce large extents. Coalescing is a small price to pay for large allocations since they happen less frequently. This reduces fragmentation while also potentially improving locality.
2017-11-16 06:48:55 +08:00
bool coalesced;
do {
assert(extent_state_get(extent) == extent_state_active);
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
extent = extent_try_coalesce_large(tsdn, arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks, rtree_ctx, eset, extent,
Eagerly coalesce large extents. Coalescing is a small price to pay for large allocations since they happen less frequently. This reduces fragmentation while also potentially improving locality.
2017-11-16 06:48:55 +08:00
&coalesced, growing_retained);
Optimize large deallocation. We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
2018-10-24 04:50:42 +08:00
} while (coalesced);
Eagerly purge oversized merged extents. This change improves memory usage slightly, at virtually no CPU cost.
2019-03-13 06:02:41 +08:00
if (extent_size_get(extent) >= oversize_threshold) {
/* Shortcut to purge the oversize extent eagerly. */
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_unlock(tsdn, &eset->mtx);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
arena_decay_extent(tsdn, arena, extent_hooks, extent);
Eagerly purge oversized merged extents. This change improves memory usage slightly, at virtually no CPU cost.
2019-03-13 06:02:41 +08:00
return;
}
Disable coalescing of cached extents. Extent splitting and coalescing is a major component of large allocation overhead, and disabling coalescing of cached extents provides a simple and effective hysteresis mechanism. Once two-phase purging is implemented, it will probably make sense to leave coalescing disabled for the first phase, but coalesce during the second phase.
2017-02-13 15:18:57 +08:00
}
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
extent_deactivate_locked(tsdn, arena, eset, extent);
Optimize extent coalescing. Refactor extent_can_coalesce(), extent_coalesce(), and extent_record() to avoid needlessly repeating extent [de]activation operations.
2017-02-13 15:00:19 +08:00
Rename extents_t -> eset_t.
2019-09-21 10:59:55 +08:00
malloc_mutex_unlock(tsdn, &eset->mtx);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_dalloc_gap(tsdn_t *tsdn, arena_t *arena, extent_t *extent) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks = arena_get_extent_hooks(arena);
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
if (extent_register(tsdn, extent)) {
extent_dalloc instead of leak when register fails. extent_register may only fail if the underlying extent and region got stolen / coalesced before we lock. Avoid doing extent_leak (which purges the region) since we don't really own the region.
2019-07-16 05:55:43 +08:00
extent_dalloc(tsdn, arena, extent);
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
return;
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_dalloc_wrapper(tsdn, arena, extent_hooks, extent);
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
}
Bypass extent_dalloc when retain is enabled. When retain is enabled, the default dalloc hook does nothing (since we avoid munmap). But the overhead preparing the call is high, specifically the extent de-register and re-register involve locking and extent / rtree modifications. Bypass the call with retain in this diff.
2018-11-01 05:54:53 +08:00
static bool
extent_may_dalloc(void) {
/* With retain enabled, the default dalloc always fails. */
return !opt_retain;
}
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
static bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_dalloc_default_impl(void *addr, size_t size) {
if (!have_dss || !extent_in_dss(addr)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent_dalloc_mmap(addr, size);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_dalloc_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
bool committed, unsigned arena_ind) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent_dalloc_default_impl(addr, size);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
static bool
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
extent_dalloc_wrapper_try(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
bool err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(extent_base_get(extent) != NULL);
assert(extent_size_get(extent) != 0);
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_addr_set(extent, extent_base_get(extent));
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
/* Try to deallocate. */
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks == &extent_hooks_default) {
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
/* Call directly to propagate tsdn. */
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
err = extent_dalloc_default_impl(extent_base_get(extent),
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
extent_size_get(extent));
} else {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
err = (extent_hooks->dalloc == NULL ||
extent_hooks->dalloc(extent_hooks,
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
extent_base_get(extent), extent_size_get(extent),
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
extent_committed_get(extent), arena_ind_get(arena)));
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!err) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_dalloc(tsdn, arena, extent);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return err;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
}
void
extent_dalloc_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent) {
Add a "dumpable" bit to the extent state. Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
2017-09-19 08:25:57 +08:00
assert(extent_dumpable_get(extent));
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Bypass extent_dalloc when retain is enabled. When retain is enabled, the default dalloc hook does nothing (since we avoid munmap). But the overhead preparing the call is high, specifically the extent de-register and re-register involve locking and extent / rtree modifications. Bypass the call with retain in this diff.
2018-11-01 05:54:53 +08:00
/* Avoid calling the default extent_dalloc unless have to. */
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default || extent_may_dalloc()) {
Bypass extent_dalloc when retain is enabled. When retain is enabled, the default dalloc hook does nothing (since we avoid munmap). But the overhead preparing the call is high, specifically the extent de-register and re-register involve locking and extent / rtree modifications. Bypass the call with retain in this diff.
2018-11-01 05:54:53 +08:00
/*
* Deregister first to avoid a race with other allocating
* threads, and reregister if deallocation fails.
*/
extent_deregister(tsdn, extent);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (!extent_dalloc_wrapper_try(tsdn, arena, extent_hooks,
Bypass extent_dalloc when retain is enabled. When retain is enabled, the default dalloc hook does nothing (since we avoid munmap). But the overhead preparing the call is high, specifically the extent de-register and re-register involve locking and extent / rtree modifications. Bypass the call with retain in this diff.
2018-11-01 05:54:53 +08:00
extent)) {
return;
}
extent_reregister(tsdn, extent);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/* Try to decommit; purge if that fails. */
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
bool zeroed;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!extent_committed_get(extent)) {
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
zeroed = true;
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
} else if (!extent_decommit_wrapper(tsdn, arena, extent_hooks, extent,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
0, extent_size_get(extent))) {
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
zeroed = true;
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
} else if (extent_hooks->purge_forced != NULL &&
!extent_hooks->purge_forced(extent_hooks,
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
extent_base_get(extent), extent_size_get(extent), 0,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_size_get(extent), arena_ind_get(arena))) {
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
zeroed = true;
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
} else if (extent_state_get(extent) == extent_state_muzzy ||
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
(extent_hooks->purge_lazy != NULL &&
!extent_hooks->purge_lazy(extent_hooks,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
extent_base_get(extent), extent_size_get(extent), 0,
extent_size_get(extent), arena_ind_get(arena)))) {
zeroed = false;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
zeroed = false;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
extent_zeroed_set(extent, zeroed);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (config_prof) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
extent_gdump_sub(tsdn, extent);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_record(tsdn, arena, extent_hooks, &arena->eset_retained,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
extent, false);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
static void
extent_destroy_default_impl(void *addr, size_t size) {
if (!have_dss || !extent_in_dss(addr)) {
pages_unmap(addr, size);
}
}
static void
extent_destroy_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
bool committed, unsigned arena_ind) {
extent_destroy_default_impl(addr, size);
}
void
extent_destroy_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent) {
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
assert(extent_base_get(extent) != NULL);
assert(extent_size_get(extent) != 0);
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
/* Deregister first to avoid a race with other allocating threads. */
extent_deregister(tsdn, extent);
extent_addr_set(extent, extent_base_get(extent));
/* Try to destroy; silently fail otherwise. */
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks == &extent_hooks_default) {
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
/* Call directly to propagate tsdn. */
extent_destroy_default_impl(extent_base_get(extent),
extent_size_get(extent));
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
} else if (extent_hooks->destroy != NULL) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks->destroy(extent_hooks,
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
extent_base_get(extent), extent_size_get(extent),
extent_committed_get(extent), arena_ind_get(arena));
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Add extent_destroy_t and use it during arena destruction. Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
2017-04-28 06:51:35 +08:00
}
extent_dalloc(tsdn, arena, extent);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_commit_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t offset, size_t length, unsigned arena_ind) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return pages_commit((void *)((uintptr_t)addr + (uintptr_t)offset),
length);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool
extent_commit_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length, bool growing_retained) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
bool err = (extent_hooks->commit == NULL ||
extent_hooks->commit(extent_hooks, extent_base_get(extent),
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
extent_size_get(extent), offset, length, arena_ind_get(arena)));
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
extent_committed_set(extent, extent_committed_get(extent) || !err);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool
extent_commit_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
return extent_commit_impl(tsdn, arena, extent_hooks, extent, offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
length, false);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_decommit_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t offset, size_t length, unsigned arena_ind) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return pages_decommit((void *)((uintptr_t)addr + (uintptr_t)offset),
length);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
bool
extent_decommit_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t length) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
WITNESS_RANK_CORE, 0);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
bool err = (extent_hooks->decommit == NULL ||
extent_hooks->decommit(extent_hooks,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_base_get(extent), extent_size_get(extent), offset, length,
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
arena_ind_get(arena)));
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Set 'committed' in extent_[de]commit_wrapper().
2016-06-06 14:24:52 +08:00
extent_committed_set(extent, extent_committed_get(extent) && err);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
#ifdef PAGES_CAN_PURGE_LAZY
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
extent_purge_lazy_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t offset, size_t length, unsigned arena_ind) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
assert(addr != NULL);
assert((offset & PAGE_MASK) == 0);
assert(length != 0);
assert((length & PAGE_MASK) == 0);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return pages_purge_lazy((void *)((uintptr_t)addr + (uintptr_t)offset),
length);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
#endif
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool
extent_purge_lazy_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length, bool growing_retained) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks->purge_lazy == NULL) {
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
return true;
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
bool err = extent_hooks->purge_lazy(extent_hooks,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_base_get(extent), extent_size_get(extent), offset, length,
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
arena_ind_get(arena));
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
return err;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool
extent_purge_lazy_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
return extent_purge_lazy_impl(tsdn, arena, extent_hooks, extent,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
offset, length, false);
}
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
#ifdef PAGES_CAN_PURGE_FORCED
static bool
extent_purge_forced_default(extent_hooks_t *extent_hooks, void *addr,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t size, size_t offset, size_t length, unsigned arena_ind) {
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
assert(addr != NULL);
assert((offset & PAGE_MASK) == 0);
assert(length != 0);
assert((length & PAGE_MASK) == 0);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return pages_purge_forced((void *)((uintptr_t)addr +
(uintptr_t)offset), length);
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
}
#endif
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool
extent_purge_forced_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length, bool growing_retained) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks->purge_forced == NULL) {
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
return true;
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
bool err = extent_hooks->purge_forced(extent_hooks,
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
extent_base_get(extent), extent_size_get(extent), offset, length,
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
arena_ind_get(arena));
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
return err;
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool
extent_purge_forced_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t offset,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
size_t length) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
return extent_purge_forced_impl(tsdn, arena, extent_hooks, extent,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
offset, length, false);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_split_default(extent_hooks_t *extent_hooks, void *addr, size_t size,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t size_a, size_t size_b, bool committed, unsigned arena_ind) {
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
if (!maps_coalesce) {
/*
* Without retain, only whole regions can be purged (required by
* MEM_RELEASE on Windows) -- therefore disallow splitting. See
* comments in extent_head_no_merge().
*/
return !opt_retain;
}
return false;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Document some of the internal extent functions.
2017-08-29 09:27:12 +08:00
/*
* Accepts the extent to split, and the characteristics of each side of the
* split. The 'a' parameters go with the 'lead' of the resulting pair of
* extents (the lower addressed portion of the split), and the 'b' parameters go
* with the trail (the higher addressed portion). This makes 'extent' the lead,
* and returns the trail (except in case of error).
*/
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static extent_t *
extent_split_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t size_a,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
szind_t szind_a, bool slab_a, size_t size_b, szind_t szind_b, bool slab_b,
bool growing_retained) {
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
assert(extent_size_get(extent) == size_a + size_b);
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks->split == NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_t *trail = extent_alloc(tsdn, arena);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (trail == NULL) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
goto label_error_a;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_init(trail, arena_ind_get(arena),
(void *)((uintptr_t)extent_base_get(extent) + size_a), size_b,
slab_b, szind_b, extent_sn_get(extent), extent_state_get(extent),
extent_zeroed_get(extent), extent_committed_get(extent),
extent_dumpable_get(extent), EXTENT_NOT_HEAD);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
rtree_leaf_elm_t *lead_elm_a, *lead_elm_b;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
{
extent_t lead;
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
extent_init(&lead, arena_ind_get(arena),
extent_addr_get(extent), size_a,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
slab_a, szind_a, extent_sn_get(extent),
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
extent_state_get(extent), extent_zeroed_get(extent),
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
extent_committed_get(extent), extent_dumpable_get(extent),
EXTENT_NOT_HEAD);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_rtree_leaf_elms_lookup(tsdn, rtree_ctx, &lead, false,
true, &lead_elm_a, &lead_elm_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_leaf_elm_t *trail_elm_a, *trail_elm_b;
extent_rtree_leaf_elms_lookup(tsdn, rtree_ctx, trail, false, true,
&trail_elm_a, &trail_elm_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
if (lead_elm_a == NULL || lead_elm_b == NULL || trail_elm_a == NULL
|| trail_elm_b == NULL) {
goto label_error_b;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_lock2(tsdn, extent, trail);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
bool err = extent_hooks->split(extent_hooks, extent_base_get(extent),
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
size_a + size_b, size_a, size_b, extent_committed_get(extent),
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
arena_ind_get(arena));
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks != &extent_hooks_default) {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Set reentrancy when invoking customized extent hooks. Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
2017-06-23 06:36:41 +08:00
}
if (err) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
goto label_error_c;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_size_set(extent, size_a);
Convert extent_t's usize to szind. Rather than storing usize only for large (and prof-promoted) allocations, store the size class index for allocations that reside within the extent, such that the size class index is valid for all extents that contain extant allocations, and invalid otherwise (mainly to make debugging simpler).
2017-03-14 08:36:57 +08:00
extent_szind_set(extent, szind_a);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_rtree_write_acquired(tsdn, lead_elm_a, lead_elm_b, extent,
szind_a, slab_a);
extent_rtree_write_acquired(tsdn, trail_elm_a, trail_elm_b, trail,
szind_b, slab_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_unlock2(tsdn, extent, trail);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return trail;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
label_error_c:
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_unlock2(tsdn, extent, trail);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
label_error_b:
extent_dalloc(tsdn, arena, trail);
label_error_a:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
extent_t *
extent_split_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *extent, size_t size_a,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
szind_t szind_a, bool slab_a, size_t size_b, szind_t szind_b, bool slab_b) {
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
return extent_split_impl(tsdn, arena, extent_hooks, extent, size_a,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
szind_a, slab_a, size_b, szind_b, slab_b, false);
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
static bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_merge_default_impl(void *addr_a, void *addr_b) {
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
if (!maps_coalesce && !opt_retain) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
if (have_dss && !extent_dss_mergeable(addr_a, addr_b)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
/*
* Returns true if the given extents can't be merged because of their head bit
* settings. Assumes the second extent has the higher address.
*/
static bool
extent_head_no_merge(extent_t *a, extent_t *b) {
assert(extent_base_get(a) < extent_base_get(b));
/*
* When coalesce is not always allowed (Windows), only merge extents
* from the same VirtualAlloc region under opt.retain (in which case
* MEM_DECOMMIT is utilized for purging).
*/
if (maps_coalesce) {
return false;
}
if (!opt_retain) {
return true;
}
/* If b is a head extent, disallow the cross-region merge. */
if (extent_is_head_get(b)) {
/*
* Additionally, sn should not overflow with retain; sanity
* check that different regions have unique sn.
*/
assert(extent_sn_comp(a, b) != 0);
return true;
}
assert(extent_sn_comp(a, b) == 0);
return false;
}
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
static bool
extent_merge_default(extent_hooks_t *extent_hooks, void *addr_a, size_t size_a,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
void *addr_b, size_t size_b, bool committed, unsigned arena_ind) {
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
if (!maps_coalesce) {
tsdn_t *tsdn = tsdn_fetch();
extent_t *a = iealloc(tsdn, addr_a);
extent_t *b = iealloc(tsdn, addr_b);
if (extent_head_no_merge(a, b)) {
return true;
}
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return extent_merge_default_impl(addr_a, addr_b);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
static bool
extent_merge_impl(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *a, extent_t *b,
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool growing_retained) {
Header refactoring: unify and de-catchall witness code.
2017-05-23 10:32:04 +08:00
witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
WITNESS_RANK_CORE, growing_retained ? 1 : 0);
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
assert(extent_base_get(a) < extent_base_get(b));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks->merge == NULL || extent_head_no_merge(a, b)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
bool err;
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
if (extent_hooks == &extent_hooks_default) {
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
/* Call directly to propagate tsdn. */
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
err = extent_merge_default_impl(extent_base_get(a),
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
extent_base_get(b));
} else {
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_pre_reentrancy(tsdn, arena);
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
err = extent_hooks->merge(extent_hooks,
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
extent_base_get(a), extent_size_get(a), extent_base_get(b),
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
extent_size_get(b), extent_committed_get(a),
arena_ind_get(arena));
Check arena in current context in pre_reentrancy.
2017-06-23 07:18:30 +08:00
extent_hook_post_reentrancy(tsdn);
Propagate tsdn to default extent hooks. This avoids bootstrapping issues for configurations that require allocation during tsd initialization. This resolves #390.
2016-06-08 04:37:22 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (err) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
/*
* The rtree writes must happen while all the relevant elements are
* owned, so the following code uses decomposed helper functions rather
* than extent_{,de}register() to do things in the right order.
*/
Disentangle arena and extent locking. Refactor arena and extent locking protocols such that arena and extent locks are never held when calling into the extent_*_wrapper() API. This requires extra care during purging since the arena lock no longer protects the inner purging logic. It also requires extra care to protect extents from being merged with adjacent extents. Convert extent_t's 'active' flag to an enumerated 'state', so that retained extents are explicitly marked as such, rather than depending on ring linkage state. Refactor the extent collections (and their synchronization) for cached and retained extents into extents_t. Incorporate LRU functionality to support purging. Incorporate page count accounting, which replaces arena->ndirty and arena->stats.retained. Assert that no core locks are held when entering any internal [de]allocation functions. This is in addition to existing assertions that no locks are held when entering external [de]allocation functions. Audit and document synchronization protocols for all arena_t fields. This fixes a potential deadlock due to recursive allocation during gdump, in a similar fashion to b49c649bc18fff4bd10a1c8adbaf1f25f6453cb6 (Fix lock order reversal during gdump.), but with a necessarily much broader code impact.
2017-01-30 13:57:14 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_elm_t *a_elm_a, *a_elm_b, *b_elm_a, *b_elm_b;
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_rtree_leaf_elms_lookup(tsdn, rtree_ctx, a, true, false, &a_elm_a,
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
&a_elm_b);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_rtree_leaf_elms_lookup(tsdn, rtree_ctx, b, true, false, &b_elm_a,
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
&b_elm_b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_lock2(tsdn, a, b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (a_elm_b != NULL) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_leaf_elm_write(tsdn, &extents_rtree, a_elm_b, NULL,
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
SC_NSIZES, false);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
if (b_elm_b != NULL) {
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_leaf_elm_write(tsdn, &extents_rtree, b_elm_a, NULL,
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
SC_NSIZES, false);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
b_elm_b = b_elm_a;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_size_set(a, extent_size_get(a) + extent_size_get(b));
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_szind_set(a, SC_NSIZES);
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
extent_sn_set(a, (extent_sn_get(a) < extent_sn_get(b)) ?
extent_sn_get(a) : extent_sn_get(b));
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_zeroed_set(a, extent_zeroed_get(a) && extent_zeroed_get(b));
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_rtree_write_acquired(tsdn, a_elm_a, b_elm_b, a, SC_NSIZES,
false);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
extent_unlock2(tsdn, a, b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
/*
* If we got here, we merged the extents; so they must be from the same
* arena (i.e. this one).
*/
assert(extent_arena_ind_get(b) == arena_ind_get(arena));
extent_dalloc(tsdn, arena, b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
}
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
bool
extent_merge_wrapper(tsdn_t *tsdn, arena_t *arena,
Extents: Eagerly initialize extent hooks. When deferred initialization was added, initializing required copying sizeof(extent_hooks_t) bytes after a pointer chase. Today, it's just a single pointer loaded from the base_t. In subsequent diffs, we'll get rid of even that.
2019-11-19 06:03:22 +08:00
extent_hooks_t *extent_hooks, extent_t *a, extent_t *b) {
return extent_merge_impl(tsdn, arena, extent_hooks, a, b, false);
Fix extent_grow_next management. Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
2017-05-27 02:06:01 +08:00
}
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
extent_boot(void) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
if (rtree_new(&extents_rtree, true)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
if (mutex_pool_init(&extent_mutex_pool, "extent_mutex_pool",
WITNESS_RANK_EXTENT_POOL)) {
return true;
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (have_dss) {
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
extent_dss_boot();
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
}
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
void
extent_util_stats_get(tsdn_t *tsdn, const void *ptr,
size_t *nfree, size_t *nregs, size_t *size) {
assert(ptr != NULL && nfree != NULL && nregs != NULL && size != NULL);
const extent_t *extent = iealloc(tsdn, ptr);
if (unlikely(extent == NULL)) {
*nfree = *nregs = *size = 0;
return;
}
*size = extent_size_get(extent);
if (!extent_slab_get(extent)) {
*nfree = 0;
*nregs = 1;
} else {
*nfree = extent_nfree_get(extent);
*nregs = bin_infos[extent_szind_get(extent)].nregs;
assert(*nfree <= *nregs);
assert(*nfree * extent_usize_get(extent) <= *size);
}
}
void
extent_util_stats_verbose_get(tsdn_t *tsdn, const void *ptr,
size_t *nfree, size_t *nregs, size_t *size,
size_t *bin_nfree, size_t *bin_nregs, void **slabcur_addr) {
assert(ptr != NULL && nfree != NULL && nregs != NULL && size != NULL
&& bin_nfree != NULL && bin_nregs != NULL && slabcur_addr != NULL);
const extent_t *extent = iealloc(tsdn, ptr);
if (unlikely(extent == NULL)) {
*nfree = *nregs = *size = *bin_nfree = *bin_nregs = 0;
*slabcur_addr = NULL;
return;
}
*size = extent_size_get(extent);
if (!extent_slab_get(extent)) {
*nfree = *bin_nfree = *bin_nregs = 0;
*nregs = 1;
*slabcur_addr = NULL;
return;
}
*nfree = extent_nfree_get(extent);
const szind_t szind = extent_szind_get(extent);
*nregs = bin_infos[szind].nregs;
assert(*nfree <= *nregs);
assert(*nfree * extent_usize_get(extent) <= *size);
Extents: Break extent-struct/arena interactions Specifically, the extent_arena_[g|s]et functions and the address randomization. These are the only things that tie the extent struct itself to the arena code.
2019-09-21 09:20:22 +08:00
const arena_t *arena = (arena_t *)atomic_load_p(
&arenas[extent_arena_ind_get(extent)], ATOMIC_RELAXED);
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
assert(arena != NULL);
const unsigned binshard = extent_binshard_get(extent);
bin_t *bin = &arena->bins[szind].bin_shards[binshard];
malloc_mutex_lock(tsdn, &bin->lock);
if (config_stats) {
*bin_nregs = *nregs * bin->stats.curslabs;
assert(*bin_nregs >= bin->stats.curregs);
*bin_nfree = *bin_nregs - bin->stats.curregs;
} else {
*bin_nfree = *bin_nregs = 0;
}
Guard slabcur fetching in extent_util
2019-10-29 00:24:42 +08:00
extent_t *slab;
if (bin->slabcur != NULL) {
slab = bin->slabcur;
} else {
slab = extent_heap_first(&bin->slabs_nonfull);
}
*slabcur_addr = slab != NULL ? extent_addr_get(slab) : NULL;
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
malloc_mutex_unlock(tsdn, &bin->lock);
}
Reference in New Issue Copy Permalink