project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
d88fa71bbd
server-skynet-source-3rd-je.../include/jemalloc/internal/tcache_externs.h

93 lines
4.0 KiB
C
Raw Normal View History

Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#ifndef JEMALLOC_INTERNAL_TCACHE_EXTERNS_H
#define JEMALLOC_INTERNAL_TCACHE_EXTERNS_H
Make headers self-contained by fixing `#include`s Header files are now self-contained, which makes the relationships between the files clearer, and crucially allows LSP tools like `clangd` to function correctly in all of our header files. I have verified that the headers are self-contained (aside from the various Windows shims) by compiling them as if they were C files – in a follow-up commit I plan to add this to CI to ensure we don't regress on this front.
2023-06-10 08:37:47 +08:00
#include "jemalloc/internal/jemalloc_preamble.h"
#include "jemalloc/internal/base.h"
#include "jemalloc/internal/cache_bin.h"
#include "jemalloc/internal/sz.h"
#include "jemalloc/internal/tcache_types.h"
Tcache: make slot sizing configurable.
2020-05-12 06:03:06 +08:00
extern bool opt_tcache;
Switch from opt.lg_tcache_max to opt.tcache_max Though for convenience, keep parsing lg_tcache_max.
2020-10-20 13:48:26 +08:00
extern size_t opt_tcache_max;
Tcache: Make size computation configurable.
2020-05-12 03:08:19 +08:00
extern ssize_t opt_lg_tcache_nslots_mul;
Tcache: make slot sizing configurable.
2020-05-12 06:03:06 +08:00
extern unsigned opt_tcache_nslots_small_min;
extern unsigned opt_tcache_nslots_small_max;
extern unsigned opt_tcache_nslots_large;
extern ssize_t opt_lg_tcache_shift;
Tcache: Make incremental gc bytes configurable.
2020-05-12 06:53:30 +08:00
extern size_t opt_tcache_gc_incr_bytes;
Tcache: Add tcache gc delay option. This can reduce flushing frequency for small size classes.
2020-05-12 07:24:17 +08:00
extern size_t opt_tcache_gc_delay_bytes;
Tcache: Make flush fractions configurable.
2020-05-14 06:32:18 +08:00
extern unsigned opt_lg_tcache_flush_small_div;
extern unsigned opt_lg_tcache_flush_large_div;
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
/*
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
* Number of tcache bins. There are SC_NBINS small-object bins, plus 0 or more
Enable per-tcache tcache_max 1. add tcache_max and nhbins into tcache_t so that they are per-tcache, with one auto tcache per thread, it's also per-thread; 2. add mallctl for each thread to set its own tcache_max (of its auto tcache); 3. store the maximum number of items in each bin instead of using a global storage; 4. add tests for the modifications above. 5. Rename `nhbins` and `tcache_maxclass` to `global_do_not_change_nhbins` and `global_do_not_change_tcache_maxclass`.
2023-08-07 02:38:30 +08:00
* large-object bins. This is only used during threads initialization and
* changing it will not reflect on initialized threads as expected. Thus,
* it should not be changed on the fly. To change the number of tcache bins
Pre-generate ncached_max for all bins for better tcache_max tuning experience.
2023-08-23 07:31:54 +08:00
* in use, refer to tcache_nbins of each tcache.
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
*/
Add mallctl to set and get ncached_max of each cache_bin. 1. `thread_tcache_ncached_max_read_sizeclass` allows users to get the ncached_max of the bin with the input sizeclass, passed in through oldp (will be upper casted if not an exact bin size is given). 2. `thread_tcache_ncached_max_write` takes in a char array representing the settings for bins in the tcache.
2023-09-20 05:37:09 +08:00
extern unsigned global_do_not_change_tcache_nbins;
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Enable per-tcache tcache_max 1. add tcache_max and nhbins into tcache_t so that they are per-tcache, with one auto tcache per thread, it's also per-thread; 2. add mallctl for each thread to set its own tcache_max (of its auto tcache); 3. store the maximum number of items in each bin instead of using a global storage; 4. add tests for the modifications above. 5. Rename `nhbins` and `tcache_maxclass` to `global_do_not_change_nhbins` and `global_do_not_change_tcache_maxclass`.
2023-08-07 02:38:30 +08:00
/*
* Maximum cached size class. Same as above, this is only used during threads
* initialization and should not be changed. To change the maximum cached size
* class, refer to tcache_max of each tcache.
*/
extern size_t global_do_not_change_tcache_maxclass;
Cache_bin: Shouldn't know about tcache. Instead, have it take the cache_bin_info_ts to use by pointer. While we're here, add a src file for the cache bin.
2020-02-26 04:14:48 +08:00
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
/*
* Explicit tcaches, managed via the tcache.{create,flush,destroy} mallctls and
* usable via the MALLOCX_TCACHE() flag. The automatic per thread tcaches are
* completely disjoint from this data structure. tcaches starts off as a sparse
* array, so it has no physical memory footprint until individual pages are
* touched. This allows the entire array to be allocated the first time an
* explicit tcache is created without a disproportionate impact on memory usage.
*/
extern tcaches_t *tcaches;
Implement use-after-free detection using junk and stash. On deallocation, sampled pointers (specially aligned) get junked and stashed into tcache (to prevent immediate reuse). The expected behavior is to have read-after-free corrupted and stopped by the junk-filling, while write-after-free is checked when flushing the stashed pointers.
2021-10-19 08:33:15 +08:00
size_t tcache_salloc(tsdn_t *tsdn, const void *ptr);
void *tcache_alloc_small_hard(tsdn_t *tsdn, arena_t *arena, tcache_t *tcache,
Fix inconsistent parameter names between definition/declaration pairs For the sake of consistency, function definitions and their corresponding declarations should use the same names for parameters. I've enabled this check in static analysis to prevent this issue from occurring again in the future.
2023-07-07 07:27:56 +08:00
cache_bin_t *cache_bin, szind_t binind, bool *tcache_success);
Tcache: split up fast and slow path data.
2020-04-08 08:48:35 +08:00
Fix inconsistent parameter names between definition/declaration pairs For the sake of consistency, function definitions and their corresponding declarations should use the same names for parameters. I've enabled this check in static analysis to prevent this issue from occurring again in the future.
2023-07-07 07:27:56 +08:00
void tcache_bin_flush_small(tsd_t *tsd, tcache_t *tcache,
cache_bin_t *cache_bin, szind_t binind, unsigned rem);
void tcache_bin_flush_large(tsd_t *tsd, tcache_t *tcache,
cache_bin_t *cache_bin, szind_t binind, unsigned rem);
void tcache_bin_flush_stashed(tsd_t *tsd, tcache_t *tcache,
cache_bin_t *cache_bin, szind_t binind, bool is_small);
Refactor the tcache initiailization 1. Pre-generate all default tcache ncached_max in tcache_boot; 2. Add getters returning default ncached_max and ncached_max_set; 3. Refactor tcache init so that it is always init with a given setting.
2023-10-18 11:17:42 +08:00
bool tcache_bin_info_default_init(const char *bin_settings_segment_cur,
size_t len_left);
Add mallctl to set and get ncached_max of each cache_bin. 1. `thread_tcache_ncached_max_read_sizeclass` allows users to get the ncached_max of the bin with the input sizeclass, passed in through oldp (will be upper casted if not an exact bin size is given). 2. `thread_tcache_ncached_max_write` takes in a char array representing the settings for bins in the tcache.
2023-09-20 05:37:09 +08:00
bool tcache_bins_ncached_max_write(tsd_t *tsd, char *settings, size_t len);
bool tcache_bin_ncached_max_read(tsd_t *tsd, size_t bin_size,
cache_bin_sz_t *ncached_max);
Implement use-after-free detection using junk and stash. On deallocation, sampled pointers (specially aligned) get junked and stashed into tcache (to prevent immediate reuse). The expected behavior is to have read-after-free corrupted and stopped by the junk-filling, while write-after-free is checked when flushing the stashed pointers.
2021-10-19 08:33:15 +08:00
void tcache_arena_reassociate(tsdn_t *tsdn, tcache_slow_t *tcache_slow,
Tcache: split up fast and slow path data.
2020-04-08 08:48:35 +08:00
tcache_t *tcache, arena_t *arena);
Integrate auto tcache into TSD. The embedded tcache is initialized upon tsd initialization. The avail arrays for the tbins will be allocated / deallocated accordingly during init / cleanup. With this change, the pointer to the auto tcache will always be available, as long as we have access to the TSD. tcache_available() (called in tcache_get()) is provided to check if we should use tcache.
2017-03-28 12:50:38 +08:00
tcache_t *tcache_create_explicit(tsd_t *tsd);
Pre-generate ncached_max for all bins for better tcache_max tuning experience.
2023-08-23 07:31:54 +08:00
void thread_tcache_max_set(tsd_t *tsd, size_t tcache_max);
Implement use-after-free detection using junk and stash. On deallocation, sampled pointers (specially aligned) get junked and stashed into tcache (to prevent immediate reuse). The expected behavior is to have read-after-free corrupted and stopped by the junk-filling, while write-after-free is checked when flushing the stashed pointers.
2021-10-19 08:33:15 +08:00
void tcache_cleanup(tsd_t *tsd);
void tcache_stats_merge(tsdn_t *tsdn, tcache_t *tcache, arena_t *arena);
bool tcaches_create(tsd_t *tsd, base_t *base, unsigned *r_ind);
void tcaches_flush(tsd_t *tsd, unsigned ind);
void tcaches_destroy(tsd_t *tsd, unsigned ind);
bool tcache_boot(tsdn_t *tsdn, base_t *base);
Tcache: split up fast and slow path data.
2020-04-08 08:48:35 +08:00
void tcache_arena_associate(tsdn_t *tsdn, tcache_slow_t *tcache_slow,
tcache_t *tcache, arena_t *arena);
Fix/refactor tcaches synchronization. Synchronize tcaches with tcaches_mtx rather than ctl_mtx. Add missing synchronization for tcache flushing. This bug was introduced by 1cb181ed632e7573fb4eab194e4d216867222d27 (Implement explicit tcache support.), which was first released in 4.0.0.
2017-01-30 13:32:39 +08:00
void tcache_prefork(tsdn_t *tsdn);
void tcache_postfork_parent(tsdn_t *tsdn);
void tcache_postfork_child(tsdn_t *tsdn);
Pass tsd to tcache_flush().
2017-06-16 06:16:18 +08:00
void tcache_flush(tsd_t *tsd);
Integrate auto tcache into TSD. The embedded tcache is initialized upon tsd initialization. The avail arrays for the tbins will be allocated / deallocated accordingly during init / cleanup. With this change, the pointer to the auto tcache will always be available, as long as we have access to the TSD. tcache_available() (called in tcache_get()) is provided to check if we should use tcache.
2017-03-28 12:50:38 +08:00
bool tsd_tcache_enabled_data_init(tsd_t *tsd);
Pre-generate ncached_max for all bins for better tcache_max tuning experience.
2023-08-23 07:31:54 +08:00
void tcache_enabled_set(tsd_t *tsd, bool enabled);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Tcache: treat small and large cache bins uniformly
2020-04-08 11:04:46 +08:00
void tcache_assert_initialized(tcache_t *tcache);
Extract out per event new wait time fetching
2020-04-16 01:49:08 +08:00
/* Only accessed by thread event. */
uint64_t tcache_gc_new_event_wait(tsd_t *tsd);
Extract out per event postponed wait time fetching
2020-04-17 04:33:56 +08:00
uint64_t tcache_gc_postponed_event_wait(tsd_t *tsd);
Push event handlers to constituent modules
2020-04-18 01:38:06 +08:00
void tcache_gc_event_handler(tsd_t *tsd, uint64_t elapsed);
Extract out per event new wait time fetching
2020-04-16 01:49:08 +08:00
uint64_t tcache_gc_dalloc_new_event_wait(tsd_t *tsd);
Extract out per event postponed wait time fetching
2020-04-17 04:33:56 +08:00
uint64_t tcache_gc_dalloc_postponed_event_wait(tsd_t *tsd);
Push event handlers to constituent modules
2020-04-18 01:38:06 +08:00
void tcache_gc_dalloc_event_handler(tsd_t *tsd, uint64_t elapsed);
Extract out per event new wait time fetching
2020-04-16 01:49:08 +08:00
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#endif /* JEMALLOC_INTERNAL_TCACHE_EXTERNS_H */
Reference in New Issue Copy Permalink