project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
856db56f6e
server-skynet-source-3rd-je.../include/jemalloc/internal/arena_externs.h

124 lines
5.5 KiB
C
Raw Normal View History

Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#ifndef JEMALLOC_INTERNAL_ARENA_EXTERNS_H
#define JEMALLOC_INTERNAL_ARENA_EXTERNS_H
Pull out arena_bin_info_t and arena_bin_t into their own file. In the process, kill arena_bin_index, which is unused. To follow are several diffs continuing this separation.
2017-10-02 08:22:06 +08:00
#include "jemalloc/internal/bin.h"
Tcache flush: keep common path state in registers. By carefully force-inlining the division constants and the operation sum count, we can eliminate redundant operations in the arena-level dalloc function. Do so.
2021-01-30 08:06:28 +08:00
#include "jemalloc/internal/div.h"
Header refactoring: unify and de-catchall extent_dss.
2017-05-24 05:36:09 +08:00
#include "jemalloc/internal/extent_dss.h"
Hooks: hook the realloc pathways that move/expand.
2018-04-24 09:07:40 +08:00
#include "jemalloc/internal/hook.h"
Header refactoring: pages.h - unify and remove from catchall.
2017-04-25 09:05:15 +08:00
#include "jemalloc/internal/pages.h"
Header refactoring: stats - unify and remove from catchall
2017-04-21 04:38:12 +08:00
#include "jemalloc/internal/stats.h"
Header refactoring: size_classes module - remove from the catchall
2017-04-20 06:09:01 +08:00
Allow PAI to calculate time until deferred work Previously the calculation of sleep time between wakeups was implemented within background_thread. This resulted in some parts of decay and hpa specific logic mixing with background thread implementation. In this change, background thread delegates this calculation to arena and it, in turn, delegates it to PAI. The next step is to implement the actual calculation of time until deferred work in HPA.
2021-08-07 05:53:05 +08:00
/*
* When the amount of pages to be purged exceeds this amount, deferred purge
* should happen.
*/
#define ARENA_DEFERRED_PURGE_NPAGES_THRESHOLD UINT64_C(1024)
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
extern ssize_t opt_dirty_decay_ms;
extern ssize_t opt_muzzy_decay_ms;
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Refactor/fix background_thread/percpu_arena bootstrapping. Refactor bootstrapping such that dlsym() is called during the bootstrapping phase that can tolerate reentrant allocation.
2017-06-01 07:45:14 +08:00
extern percpu_arena_mode_t opt_percpu_arena;
Make eligible global variables `static` and/or `const` For better or worse, Jemalloc has a significant number of global variables. Making all eligible global variables `static` and/or `const` at least makes it slightly easier to reason about them, as these qualifications communicate to the programmer restrictions on their use without having to `grep` the whole codebase.
2023-05-19 01:22:58 +08:00
extern const char *const percpu_arena_mode_names[];
Implement per-CPU arena. The new feature, opt.percpu_arena, determines thread-arena association dynamically based CPU id. Three modes are supported: "percpu", "phycpu" and disabled. "percpu" uses the current core id (with help from sched_getcpu()) directly as the arena index, while "phycpu" will assign threads on the same physical CPU to the same arena. In other words, "percpu" means # of arenas == # of CPUs, while "phycpu" has # of arenas == 1/2 * (# of CPUs). Note that no runtime check on whether hyper threading is enabled is added yet. When enabled, threads will be migrated between arenas when a CPU change is detected. In the current design, to reduce overhead from reading CPU id, each arena tracks the thread accessed most recently. When a new thread comes in, we will read CPU id and update arena if necessary.
2017-02-03 09:02:05 +08:00
Tcache flush: keep common path state in registers. By carefully force-inlining the division constants and the operation sum count, we can eliminate redundant operations in the arena-level dalloc function. Do so.
2021-01-30 08:06:28 +08:00
extern div_info_t arena_binind_div_info[SC_NBINS];
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
extern emap_t arena_emap_global;
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
Rename huge_threshold to oversize_threshold. The keyword huge tend to remind people of huge pages which is not relevent to the feature.
2019-01-25 08:15:04 +08:00
extern size_t opt_oversize_threshold;
extern size_t oversize_threshold;
Implement huge arena: opt.huge_threshold. The feature allows using a dedicated arena for huge allocations. We want the addtional arena to separate huge allocation because: 1) mixing small extents with huge ones causes fragmentation over the long run (this feature reduces VM size significantly); 2) with many arenas, huge extents rarely get reused across threads; and 3) huge allocations happen way less frequently, therefore no concerns for lock contention.
2018-05-22 04:33:48 +08:00
Arena: share bin offsets in a global. This saves us a cache miss when lookup up the arena bin offset in a remote arena during tcache flush. All arenas share the base offset, and so we don't need to look it up repeatedly for each arena. Secondarily, it shaves 288 bytes off the arena on, e.g., x86-64.
2021-01-31 07:35:33 +08:00
/*
* arena_bin_offsets[binind] is the offset of the first bin shard for size class
* binind.
*/
extern uint32_t arena_bin_offsets[SC_NBINS];
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_basic_stats_merge(tsdn_t *tsdn, arena_t *arena,
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
unsigned *nthreads, const char **dss, ssize_t *dirty_decay_ms,
ssize_t *muzzy_decay_ms, size_t *nactive, size_t *ndirty, size_t *nmuzzy);
Convert arena->stats synchronization to atomics.
2017-02-13 09:43:33 +08:00
void arena_stats_merge(tsdn_t *tsdn, arena_t *arena, unsigned *nthreads,
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
const char **dss, ssize_t *dirty_decay_ms, ssize_t *muzzy_decay_ms,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
size_t *nactive, size_t *ndirty, size_t *nmuzzy, arena_stats_t *astats,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
bin_stats_data_t *bstats, arena_stats_large_t *lstats,
PA: Use an SEC in fron of the HPA shard.
2020-10-17 04:14:59 +08:00
pac_estats_t *estats, hpa_shard_stats_t *hpastats, sec_stats_t *secstats);
Wake up background threads on demand This change allows every allocator conforming to PAI communicate that it deferred some work for the future. Without it if a background thread goes into indefinite sleep, there is no way to notify it about upcoming deferred work.
2021-08-19 10:24:37 +08:00
void arena_handle_deferred_work(tsdn_t *tsdn, arena_t *arena);
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
edata_t *arena_extent_alloc_large(tsdn_t *tsdn, arena_t *arena,
PA: Take zero as a bool rather than as a bool *. Now that we've moved junking to a higher level of the allocation stack, we don't care about this performance optimization (which only occurred in debug modes).
2020-03-15 09:10:29 +08:00
size_t usize, size_t alignment, bool zero);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_extent_dalloc_large_prep(tsdn_t *tsdn, arena_t *arena,
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
edata_t *edata);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_extent_ralloc_large_shrink(tsdn_t *tsdn, arena_t *arena,
Fix inconsistent parameter names between definition/declaration pairs For the sake of consistency, function definitions and their corresponding declarations should use the same names for parameters. I've enabled this check in static analysis to prevent this issue from occurring again in the future.
2023-07-07 07:27:56 +08:00
edata_t *edata, size_t oldusize);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_extent_ralloc_large_expand(tsdn_t *tsdn, arena_t *arena,
Fix inconsistent parameter names between definition/declaration pairs For the sake of consistency, function definitions and their corresponding declarations should use the same names for parameters. I've enabled this check in static analysis to prevent this issue from occurring again in the future.
2023-07-07 07:27:56 +08:00
edata_t *edata, size_t oldusize);
PAC: Move in decay rate setting.
2020-06-04 09:30:33 +08:00
bool arena_decay_ms_set(tsdn_t *tsdn, arena_t *arena, extent_state_t state,
ssize_t decay_ms);
ssize_t arena_decay_ms_get(arena_t *arena, extent_state_t state);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_decay(tsdn_t *tsdn, arena_t *arena, bool is_background_thread,
bool all);
Allow PAI to calculate time until deferred work Previously the calculation of sleep time between wakeups was implemented within background_thread. This resulted in some parts of decay and hpa specific logic mixing with background thread implementation. In this change, background thread delegates this calculation to arena and it, in turn, delegates it to PAI. The next step is to implement the actual calculation of time until deferred work in HPA.
2021-08-07 05:53:05 +08:00
uint64_t arena_time_until_deferred(tsdn_t *tsdn, arena_t *arena);
Add HPA deferral functionality.
2021-06-05 07:07:27 +08:00
void arena_do_deferred_work(tsdn_t *tsdn, arena_t *arena);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_reset(tsd_t *tsd, arena_t *arena);
void arena_destroy(tsd_t *tsd, arena_t *arena);
Arena: fill in terms of cache_bins.
2020-04-08 08:49:50 +08:00
void arena_cache_bin_fill_small(tsdn_t *tsdn, arena_t *arena,
cache_bin_t *cache_bin, cache_bin_info_t *cache_bin_info, szind_t binind,
const unsigned nfill);
Stop depending on JEMALLOC_N() for function interception during testing. Instead, always define function pointers for interceptable functions, but mark them const unless testing, so that the compiler can optimize out the pointer dereferences.
2017-05-02 14:10:42 +08:00
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void *arena_malloc_hard(tsdn_t *tsdn, arena_t *arena, size_t size,
Reduce the memory overhead of sampled small allocations Previously, small allocations which were sampled as part of heap profiling were rounded up to `SC_LARGE_MINCLASS`. This additional memory usage becomes problematic when the page size is increased, as noted in #2358. Small allocations are now rounded up to the nearest multiple of `PAGE` instead, reducing the memory overhead by a factor of 4 in the most extreme cases.
2023-06-03 06:15:37 +08:00
szind_t ind, bool zero, bool slab);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void *arena_palloc(tsdn_t *tsdn, arena_t *arena, size_t usize,
Reduce the memory overhead of sampled small allocations Previously, small allocations which were sampled as part of heap profiling were rounded up to `SC_LARGE_MINCLASS`. This additional memory usage becomes problematic when the page size is increased, as noted in #2358. Small allocations are now rounded up to the nearest multiple of `PAGE` instead, reducing the memory overhead by a factor of 4 in the most extreme cases.
2023-06-03 06:15:37 +08:00
size_t alignment, bool zero, bool slab, tcache_t *tcache);
void arena_prof_promote(tsdn_t *tsdn, void *ptr, size_t usize,
size_t bumped_usize);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_dalloc_promoted(tsdn_t *tsdn, void *ptr, tcache_t *tcache,
Push down iealloc() calls. Call iealloc() as deep into call chains as possible without causing redundant calls.
2017-03-21 02:00:07 +08:00
bool slow_path);
Rework the bin locking around tcache refill / flush. Previously, tcache fill/flush (as well as small alloc/dalloc on the arena) may potentially drop the bin lock for slab_alloc and slab_dalloc. This commit refactors the logic so that the slab calls happen in the same function / level as the bin lock / unlock. The main purpose is to be able to use flat combining without having to keep track of stack state. In the meantime, this change reduces the locking, especially for slab_dalloc calls, where nothing happens after the call.
2020-02-08 06:53:36 +08:00
void arena_slab_dalloc(tsdn_t *tsdn, arena_t *arena, edata_t *slab);
Tcache flush: keep common path state in registers. By carefully force-inlining the division constants and the operation sum count, we can eliminate redundant operations in the arena-level dalloc function. Do so.
2021-01-30 08:06:28 +08:00
void arena_dalloc_bin_locked_handle_newly_empty(tsdn_t *tsdn, arena_t *arena,
edata_t *slab, bin_t *bin);
void arena_dalloc_bin_locked_handle_newly_nonempty(tsdn_t *tsdn, arena_t *arena,
edata_t *slab, bin_t *bin);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_dalloc_small(tsdn_t *tsdn, void *ptr);
Push down iealloc() calls. Call iealloc() as deep into call chains as possible without causing redundant calls.
2017-03-21 02:00:07 +08:00
bool arena_ralloc_no_move(tsdn_t *tsdn, void *ptr, size_t oldsize, size_t size,
Optimize ixalloc by avoiding a size lookup.
2018-06-05 04:36:06 +08:00
size_t extra, bool zero, size_t *newsize);
Push down iealloc() calls. Call iealloc() as deep into call chains as possible without causing redundant calls.
2017-03-21 02:00:07 +08:00
void *arena_ralloc(tsdn_t *tsdn, arena_t *arena, void *ptr, size_t oldsize,
Reduce the memory overhead of sampled small allocations Previously, small allocations which were sampled as part of heap profiling were rounded up to `SC_LARGE_MINCLASS`. This additional memory usage becomes problematic when the page size is increased, as noted in #2358. Small allocations are now rounded up to the nearest multiple of `PAGE` instead, reducing the memory overhead by a factor of 4 in the most extreme cases.
2023-06-03 06:15:37 +08:00
size_t size, size_t alignment, bool zero, bool slab, tcache_t *tcache,
Hooks: hook the realloc pathways that move/expand.
2018-04-24 09:07:40 +08:00
hook_ralloc_args_t *hook_args);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
dss_prec_t arena_dss_prec_get(arena_t *arena);
Move extent ehook calls into ehooks
2019-12-03 06:19:22 +08:00
ehooks_t *arena_get_ehooks(arena_t *arena);
Move extent hook getters/setters to arena.c This is where they're logically scoped; they access arena data.
2019-11-19 04:59:34 +08:00
extent_hooks_t *arena_set_extent_hooks(tsd_t *tsd, arena_t *arena,
extent_hooks_t *extent_hooks);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
bool arena_dss_prec_set(arena_t *arena, dss_prec_t dss_prec);
Add arena-level name. An arena-level name can help identify manual arenas.
2022-09-02 07:42:56 +08:00
void arena_name_get(arena_t *arena, char *name);
void arena_name_set(arena_t *arena, const char *name);
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
ssize_t arena_dirty_decay_ms_default_get(void);
bool arena_dirty_decay_ms_default_set(ssize_t decay_ms);
ssize_t arena_muzzy_decay_ms_default_get(void);
bool arena_muzzy_decay_ms_default_set(ssize_t decay_ms);
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
bool arena_retain_grow_limit_get_set(tsd_t *tsd, arena_t *arena,
size_t *old_limit, size_t *new_limit);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
unsigned arena_nthreads_get(arena_t *arena, bool internal);
void arena_nthreads_inc(arena_t *arena, bool internal);
void arena_nthreads_dec(arena_t *arena, bool internal);
add experimental.arenas_create_ext mallctl This mallctl accepts an arena_config_t structure which can be used to customize the behavior of the arena. Right now it contains extent_hooks and a new option, metadata_use_hooks, which controls whether the extent hooks are also used for metadata allocation. The medata_use_hooks option has two main use cases: 1. In heterogeneous memory systems, to avoid metadata being placed on potentially slower memory. 2. Avoiding virtual memory from being leaked as a result of metadata allocation failure originating in an extent hook.
2021-08-23 20:03:35 +08:00
arena_t *arena_new(tsdn_t *tsdn, unsigned ind, const arena_config_t *config);
Fix the arena selection for oversized allocations. Use the per-arena oversize_threshold, instead of the global setting.
2023-06-07 04:37:07 +08:00
bool arena_init_huge(arena_t *a0);
Avoid creating bg thds for huge arena lone. For low arena count settings, the huge threshold feature may trigger an unwanted bg thd creation. Given that the huge arena does eager purging by default, bypass bg thd creation when initializing the huge arena.
2019-01-15 06:16:09 +08:00
bool arena_is_huge(unsigned arena_ind);
Implement huge arena: opt.huge_threshold. The feature allows using a dedicated arena for huge allocations. We want the addtional arena to separate huge allocation because: 1) mixing small extents with huge ones causes fragmentation over the long run (this feature reduces VM size significantly); 2) with many arenas, huge extents rarely get reused across threads; and 3) huge allocations happen way less frequently, therefore no concerns for lock contention.
2018-05-22 04:33:48 +08:00
arena_t *arena_choose_huge(tsd_t *tsd);
Extract bin shard selection out of bin locking
2020-04-23 08:22:43 +08:00
bin_t *arena_bin_choose(tsdn_t *tsdn, arena_t *arena, szind_t binind,
Add support for sharded bins within an arena. This makes it possible to have multiple set of bins in an arena, which improves arena scalability because the bins (especially the small ones) are always the limiting factor in production workload. A bin shard is picked on allocation; each extent tracks the bin shard id for deallocation. The shard size will be determined using runtime options.
2018-11-13 07:56:04 +08:00
unsigned *binshard);
Add batch allocation from fresh slabs
2020-04-23 09:13:06 +08:00
size_t arena_fill_small_fresh(tsdn_t *tsdn, arena_t *arena, szind_t binind,
Add zero option to arena batch allocation
2020-07-23 08:01:44 +08:00
void **ptrs, size_t nfill, bool zero);
HPA: Introduce a redesigned hpa_central_t. For now, this only handles allocating virtual address space to shards, with no reuse. This is framework, though; it will change over time.
2021-05-08 04:54:26 +08:00
bool arena_boot(sc_data_t *sc_data, base_t *base, bool hpa);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_prefork0(tsdn_t *tsdn, arena_t *arena);
void arena_prefork1(tsdn_t *tsdn, arena_t *arena);
void arena_prefork2(tsdn_t *tsdn, arena_t *arena);
void arena_prefork3(tsdn_t *tsdn, arena_t *arena);
void arena_prefork4(tsdn_t *tsdn, arena_t *arena);
void arena_prefork5(tsdn_t *tsdn, arena_t *arena);
void arena_prefork6(tsdn_t *tsdn, arena_t *arena);
Add extent_grow_mtx in pre_ / post_fork handlers. This fixed the issue that could cause the child process to stuck after fork.
2017-06-30 07:01:35 +08:00
void arena_prefork7(tsdn_t *tsdn, arena_t *arena);
PA: Use an SEC in fron of the HPA shard.
2020-10-17 04:14:59 +08:00
void arena_prefork8(tsdn_t *tsdn, arena_t *arena);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
void arena_postfork_parent(tsdn_t *tsdn, arena_t *arena);
void arena_postfork_child(tsdn_t *tsdn, arena_t *arena);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#endif /* JEMALLOC_INTERNAL_ARENA_EXTERNS_H */
Reference in New Issue Copy Permalink