server-skynet-source-3rd-je.../include/jemalloc/internal
David Goldblatt da63f23e68 HPA: Track pending purges/hugifies in the psset.
This finishes the refactoring of the HPA/psset interactions the past few commits
have been building towards.

Rather than the HPA removing and then reinserting hpdatas, it simply begins
updates and ends them.  These updates can set flags on the hpdata that prevent
it from being returned for certain types of requests.  For example, it can call
hpdata_alloc_allowed_set(hpdata, false) during an update, at which point the
given hpdata will no longer be returned for psset_pick_alloc requests.

This has various of benefits:
- It maintains stats correctness during purges and hugifies.
- It allows simpler and more explicit concurrency control for the various
  special cases (e.g. allocations are disallowed during purge, but not during
  hugify).
- It lets allocations and deallocations avoid disturbing the purging and
  hugification orderings.  If an hpdata "loses its place" in one of the queues
  just do to an alloc / dalloc, it can result in pathological edge cases where
  very hot, very full hugepages never get hugified  (and cold extents on the
  same hugepage as hot ones never get purged).

The key benefit though is that tracking hpdatas to be purged / hugified in a
principled way will let us do delayed purging and hugification.  Eventually this
will let us move these operations to background threads, but in the short term
the benefit is that it will let us have global purging policies (e.g. purge when
the entire arena has too many dirty pages, rather than any particular hugepage).
2021-02-04 20:58:31 -08:00
..
activity_callback.h Add experimental.thread.activity_callback. 2020-11-05 12:33:25 -08:00
arena_externs.h Arena: share bin offsets in a global. 2021-02-04 14:10:43 -08:00
arena_inlines_a.h Cache an arena's index in the arena. 2021-02-04 14:10:43 -08:00
arena_inlines_b.h Use ticker_geom_t for arena tcache decay. 2021-02-04 14:10:43 -08:00
arena_stats.h Minor update to locked int 2020-05-12 08:23:15 -07:00
arena_structs.h Use ticker_geom_t for arena tcache decay. 2021-02-04 14:10:43 -08:00
arena_types.h Use ticker_geom_t for arena tcache decay. 2021-02-04 14:10:43 -08:00
assert.h Disentangle assert and util 2017-03-06 15:08:43 -08:00
atomic_c11.h Introduce a backport of C11 atomics 2017-03-03 13:40:59 -08:00
atomic_gcc_atomic.h Workaround to address g++ unused variable warnings 2019-07-30 11:37:56 -07:00
atomic_gcc_sync.h Restrict lwsync to powerpc64 only 2018-10-24 11:18:55 -07:00
atomic_msvc.h Introduce a backport of C11 atomics 2017-03-03 13:40:59 -08:00
atomic.h atomic: add atomic_load_sub_store 2020-07-09 13:41:04 -07:00
background_thread_externs.h Introduce decay module and put decay objects in PA 2020-04-10 13:12:47 -07:00
background_thread_inlines.h PAC: Move in decay. 2020-07-09 13:41:04 -07:00
background_thread_structs.h Add max_per_bg_thd stats for per background thread mutexes. 2019-09-13 09:23:57 -07:00
base_structs.h Move relevant index into the ehooks_t itself. 2019-12-20 10:18:40 -08:00
base.h Rtree: take the base allocator as a parameter. 2020-02-18 11:22:09 -08:00
bin_info.h Extents: Split out extent struct. 2019-09-23 23:06:27 -07:00
bin_stats.h Reduce footprint of bin_t. 2019-11-21 11:08:36 -08:00
bin_types.h Rename extent_t to edata_t. 2019-12-20 10:18:40 -08:00
bin.h Bin: Move stats closer to the mutex. 2021-02-04 14:10:43 -08:00
bit_util.h bit_util: Guarantee popcount's presence. 2020-12-07 06:21:08 -08:00
bitmap.h bit_util: Change ffs functions indexing. 2020-07-30 15:25:23 -07:00
buf_writer.h Unify write callback signature 2020-05-11 14:51:24 -07:00
cache_bin.h Cache bin: Don't reverse flush order. 2021-02-04 14:10:43 -08:00
ckh.h Header refactoring: tsd - cleanup and dependency breaking. 2017-05-01 10:49:56 -07:00
counter.h Migrate counter to use locked int 2020-05-12 08:23:15 -07:00
ctl.h Optimize stats print using partial name-to-mib 2020-12-18 10:39:58 -08:00
decay.h Decay: Take current time as an argument. 2020-04-10 13:12:47 -07:00
div.h Add the div module, which allows fast division by dynamic values. 2017-12-21 14:25:43 -08:00
ecache.h Rename ecache_grow -> geom_grow. 2020-08-19 16:53:21 -07:00
edata_cache.h Rename edata_tree_t -> edata_avail_t. 2020-11-13 13:42:11 -08:00
edata.h Record request size in prof info 2021-01-07 20:39:49 -08:00
ehooks.h Ehooks: Fix a build warning. 2020-02-25 10:21:03 -08:00
emap.h Tcache fluhing: increase cache miss parallelism. 2021-02-04 14:10:43 -08:00
emitter.h Unify write callback signature 2020-05-11 14:51:24 -07:00
eset.h Edata: split up different list linkage uses. 2020-07-08 13:20:59 -07:00
exp_grow.h Rename geom_grow -> exp_grow. 2020-11-13 13:42:33 -08:00
extent_dss.h Header refactoring: unify and de-catchall extent_dss. 2017-05-31 13:08:45 -07:00
extent_mmap.h Header refactoring: unify and de-catchall extent_mmap module. 2017-05-31 13:08:45 -07:00
extent.h PA->PAC: Make extent.c forget about PA. 2020-07-09 13:41:04 -07:00
flat_bitmap.h flat bitmap: Add a bitwise and/or/not. 2021-02-04 20:58:31 -08:00
fxp.h Add fxp: A fixed-point math library. 2020-12-04 23:48:19 -08:00
hash.h Fix Undefined Behavior in hash.h 2020-02-13 12:25:26 -08:00
hook.h Hooks: Protect against reentrancy. 2018-05-18 11:43:03 -07:00
hpa_central.h Use the edata_cache_small_t in the HPA. 2020-11-05 12:34:43 -08:00
hpa.h Move empty slab tracking to the psset. 2021-02-04 20:58:31 -08:00
hpdata.h HPA: Track pending purges/hugifies in the psset. 2021-02-04 20:58:31 -08:00
inspect.h Extents: Split out introspection functionality. 2019-12-20 10:18:40 -08:00
jemalloc_internal_decls.h DragonFlyBSD build support. 2020-10-27 12:35:19 -07:00
jemalloc_internal_defs.h.in Mac OS: Tag mapped pages. 2021-02-03 15:05:53 -08:00
jemalloc_internal_externs.h Use ticker_geom_t for arena tcache decay. 2021-02-04 14:10:43 -08:00
jemalloc_internal_includes.h Consolidate prof inline function headers 2020-05-11 12:24:56 -07:00
jemalloc_internal_inlines_a.h Use ticker_geom_t for arena tcache decay. 2021-02-04 14:10:43 -08:00
jemalloc_internal_inlines_b.h Tcache: split up fast and slow path data. 2020-04-14 15:20:19 -07:00
jemalloc_internal_inlines_c.h Emap: Standardize naming. 2020-02-17 10:50:51 -08:00
jemalloc_internal_macros.h Use __forceinline for JEMALLOC_ALWAYS_INLINE on msvc 2019-11-12 13:50:25 -08:00
jemalloc_internal_types.h Unify write callback signature 2020-05-11 14:51:24 -07:00
jemalloc_preamble.h.in utrace support with label based signature. 2020-11-30 11:43:00 -08:00
large_externs.h Record request size in prof info 2021-01-07 20:39:49 -08:00
lockedint.h Migrate counter to use locked int 2020-05-12 08:23:15 -07:00
log.h Logging: capitalize log macro. 2017-10-02 20:44:43 -07:00
malloc_io.h Unify write callback signature 2020-05-11 14:51:24 -07:00
mutex_pool.h Header refactoring: unify and de-catchall mutex_pool. 2017-05-31 13:08:45 -07:00
mutex_prof.h Add prof stats mutex stats 2021-01-07 20:39:49 -08:00
mutex.h Add max_per_bg_thd stats for per background thread mutexes. 2019-09-13 09:23:57 -07:00
nstime.h High Resolution Timestamps for Profiling 2020-06-15 12:12:49 -07:00
pa.h HPA: Manage whole hugepages at a time. 2020-12-07 06:21:08 -08:00
pac.h Add a per-arena oversize_threshold. 2020-11-13 13:45:35 -08:00
pages.h Introduce hpdata_t. 2020-12-07 06:21:08 -08:00
pai.h Introduce PAI: Page allocator interface 2020-07-09 13:41:04 -07:00
peak_event.h Add thread.peak.[read|reset] mallctls. 2020-06-11 13:54:22 -07:00
peak.h Add peak_t, for tracking allocator net max. 2020-06-11 13:54:22 -07:00
ph.h Add any() and remove_any() to ph. 2017-03-07 10:25:33 -08:00
private_namespace.sh Automatically generate private symbol name mangling macros. 2017-05-11 23:06:54 -07:00
private_symbols.sh Automatically generate private symbol name mangling macros. 2017-05-11 23:06:54 -07:00
prng.h PRNG: Remove atomic functionality. 2020-10-05 19:55:57 -07:00
prof_data.h Do not create size class tables for non-prof builds 2020-08-24 20:10:02 -07:00
prof_externs.h Track per size class internal fragmentation 2021-01-07 20:39:49 -08:00
prof_inlines.h Record request size in prof info 2021-01-07 20:39:49 -08:00
prof_log.h Relocate a few prof utilities to the right modules 2020-06-29 14:27:50 -07:00
prof_recent.h Correct usize in prof last-N record 2020-09-09 13:31:35 -07:00
prof_stats.h Track per size class internal fragmentation 2021-01-07 20:39:49 -08:00
prof_structs.h Record request size in prof info 2021-01-07 20:39:49 -08:00
prof_sys.h Move unwind to prof_sys 2020-06-29 14:27:50 -07:00
prof_types.h Do not create size class tables for non-prof builds 2020-08-24 20:10:02 -07:00
psset.h HPA: Track pending purges/hugifies in the psset. 2021-02-04 20:58:31 -08:00
public_namespace.sh Replace tabs following #define with spaces. 2017-01-20 21:45:53 -08:00
public_unnamespace.sh Fix name mangling for stress tests. 2014-01-16 17:38:01 -08:00
ql.h QL, QR: Add documentation. 2020-04-11 10:32:11 -07:00
qr.h QL, QR: Add documentation. 2020-04-11 10:32:11 -07:00
quantum.h LQ_QUANTUM should be 4 on mips64 hardware. 2020-05-14 10:30:37 -07:00
rb.h Header refactoring: break out rb.h dependencies 2017-04-11 11:52:30 -07:00
rtree_tsd.h Clean compilation -Wextra 2018-07-09 21:40:42 -07:00
rtree.h Rtree: Remove the per-field accessors. 2020-04-10 13:12:47 -07:00
safety_check.h Add safety check on sdallocx slow / sampled path. 2020-01-31 00:04:22 -08:00
sc.h Avoid a -Wundef warning on LG_SLAB_MAXREGS. 2020-09-17 10:05:40 -07:00
sec.h Add SEC module: a small extent cache. 2020-10-23 11:14:34 -07:00
seq.h Add the Seq module, a simple seqlock implementation. 2018-05-18 11:43:03 -07:00
slab_data.h Rearrange slab data and constants 2019-09-23 23:06:27 -07:00
smoothstep.h Cleanup smoothstep.sh / .h. 2017-05-25 16:52:10 -07:00
smoothstep.sh Cleanup smoothstep.sh / .h. 2017-05-25 16:52:10 -07:00
spin.h Power: disable the CPU_SPINWAIT macro. 2017-10-04 18:37:23 -07:00
stats.h HPA: Add stats for the hpa_shard. 2020-10-23 11:14:34 -07:00
sz.h Do not bump to large size for page aligned request 2020-12-29 17:09:58 -08:00
tcache_externs.h Switch from opt.lg_tcache_max to opt.tcache_max 2020-10-22 20:40:41 -07:00
tcache_inlines.h Allow opt.tcache_max to accept small size classes. 2020-10-24 20:43:44 -07:00
tcache_structs.h Add a hard limit on tcache max size class. 2020-10-16 13:49:51 -07:00
tcache_types.h Add a hard limit on tcache max size class. 2020-10-16 13:49:51 -07:00
test_hooks.h Rename hooks module to test_hooks. 2018-05-18 11:43:03 -07:00
thread_event.h Add batch allocation API 2020-07-31 09:16:50 -07:00
ticker.h Add ticker_geom_t. 2021-02-04 14:10:43 -08:00
tsd_generic.h Fix recursive malloc during bootstrap on QNX 2020-12-18 10:05:59 -08:00
tsd_malloc_thread_cleanup.h Enforce TLS_MODEL attribute. 2019-04-16 11:07:15 -07:00
tsd_tls.h Enforce TLS_MODEL attribute. 2019-04-16 11:07:15 -07:00
tsd_types.h Header refactoring: tsd - cleanup and dependency breaking. 2017-05-01 10:49:56 -07:00
tsd_win.h Header refactoring: tsd - cleanup and dependency breaking. 2017-05-01 10:49:56 -07:00
tsd.h Use ticker_geom_t for arena tcache decay. 2021-02-04 14:10:43 -08:00
typed_list.h Add empty test and concat for typed list 2020-07-31 09:16:50 -07:00
util.h Tcache flush: prefetch edata contents. 2021-02-04 14:10:43 -08:00
witness.h Track per size class internal fragmentation 2021-01-07 20:39:49 -08:00