project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
527dd4cdb8
server-skynet-source-3rd-je.../src/ctl.c

3596 lines
104 KiB
C
Raw Normal View History

Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define JEMALLOC_CTL_C_
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
#include "jemalloc/internal/jemalloc_preamble.h"
#include "jemalloc/internal/jemalloc_internal_includes.h"
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Header refactoring: move assert.h out of the catch-all
2017-04-12 05:43:12 +08:00
#include "jemalloc/internal/assert.h"
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
#include "jemalloc/internal/ctl.h"
Header refactoring: unify and de-catchall extent_dss.
2017-05-24 05:36:09 +08:00
#include "jemalloc/internal/extent_dss.h"
Header refactoring: unify and de-catchall extent_mmap module.
2017-05-24 05:42:32 +08:00
#include "jemalloc/internal/extent_mmap.h"
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
#include "jemalloc/internal/inspect.h"
Header refactoring: unify and de-catchall mutex module
2017-05-24 03:28:19 +08:00
#include "jemalloc/internal/mutex.h"
Header refactoring: unify nstime.h and move it out of the catch-all
2017-04-18 07:17:02 +08:00
#include "jemalloc/internal/nstime.h"
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
#include "jemalloc/internal/sc.h"
Header refactoring: move util.h out of the catchall
2017-04-12 04:31:16 +08:00
#include "jemalloc/internal/util.h"
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/******************************************************************************/
/* Data. */
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
/*
* ctl_mtx protects the following:
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
* - ctl_stats->*
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
*/
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static malloc_mutex_t ctl_mtx;
static bool ctl_initialized;
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
static ctl_stats_t *ctl_stats;
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
static ctl_arenas_t *ctl_arenas;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
/******************************************************************************/
/* Helpers for named and indexed nodes. */
Get rid of most of the various inline macros.
2017-04-22 00:37:34 +08:00
static const ctl_named_node_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_named_node(const ctl_node_t *node) {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
return ((node->named) ? (const ctl_named_node_t *)node : NULL);
}
Get rid of most of the various inline macros.
2017-04-22 00:37:34 +08:00
static const ctl_named_node_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_named_children(const ctl_named_node_t *node, size_t index) {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
const ctl_named_node_t *children = ctl_named_node(node->children);
return (children ? &children[index] : NULL);
}
Get rid of most of the various inline macros.
2017-04-22 00:37:34 +08:00
static const ctl_indexed_node_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_indexed_node(const ctl_node_t *node) {
Convert to uniform style: cond == false --> !cond
2014-10-04 01:16:09 +08:00
return (!node->named ? (const ctl_indexed_node_t *)node : NULL);
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/******************************************************************************/
/* Function prototypes for non-inline static functions. */
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_PROTO(n) \
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
static int n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, \
void *oldp, size_t *oldlenp, void *newp, size_t newlen);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define INDEX_PROTO(n) \
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
static const ctl_named_node_t *n##_index(tsdn_t *tsdn, \
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
const size_t *mib, size_t miblen, size_t i);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Add release versioning support. Base version string on 'git describe --long', and provide cpp macros in jemalloc.h. Add the version mallctl.
2010-03-03 05:01:16 +08:00
CTL_PROTO(version)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(epoch)
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
CTL_PROTO(background_thread)
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
CTL_PROTO(max_background_threads)
Add the "thread.tcache.enabled" mallctl.
2012-03-27 09:54:44 +08:00
CTL_PROTO(thread_tcache_enabled)
Rename the "tcache.flush" mallctl to "thread.tcache.flush".
2012-03-17 08:09:32 +08:00
CTL_PROTO(thread_tcache_flush)
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
CTL_PROTO(thread_prof_name)
CTL_PROTO(thread_prof_active)
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
CTL_PROTO(thread_arena)
Add per thread allocation counters, and enhance heap sampling. Add the "thread.allocated" and "thread.deallocated" mallctls, which can be used to query the total number of bytes ever allocated/deallocated by the calling thread. Add s2u() and sa2u(), which can be used to compute the usable size that will result from an allocation request of a particular size/alignment. Re-factor ipalloc() to use sa2u(). Enhance the heap profiler to trigger samples based on usable size, rather than request size. This has a subtle, but important, impact on the accuracy of heap sampling. For example, previous to this change, 16- and 17-byte objects were sampled at nearly the same rate, but 17-byte objects actually consume 32 bytes each. Therefore it was possible for the sample to be somewhat skewed compared to actual memory usage of the allocated objects.
2010-10-21 08:39:18 +08:00
CTL_PROTO(thread_allocated)
Add the "thread.[de]allocatedp" mallctl's.
2010-12-04 07:55:47 +08:00
CTL_PROTO(thread_allocatedp)
Add per thread allocation counters, and enhance heap sampling. Add the "thread.allocated" and "thread.deallocated" mallctls, which can be used to query the total number of bytes ever allocated/deallocated by the calling thread. Add s2u() and sa2u(), which can be used to compute the usable size that will result from an allocation request of a particular size/alignment. Re-factor ipalloc() to use sa2u(). Enhance the heap profiler to trigger samples based on usable size, rather than request size. This has a subtle, but important, impact on the accuracy of heap sampling. For example, previous to this change, 16- and 17-byte objects were sampled at nearly the same rate, but 17-byte objects actually consume 32 bytes each. Therefore it was possible for the sample to be somewhat skewed compared to actual memory usage of the allocated objects.
2010-10-21 08:39:18 +08:00
CTL_PROTO(thread_deallocated)
Add the "thread.[de]allocatedp" mallctl's.
2010-12-04 07:55:47 +08:00
CTL_PROTO(thread_deallocatedp)
Add "thread.idle" mallctl. This can encapsulate various internal cleaning logic, and can be used to free up resources before a long sleep.
2020-01-23 03:13:26 +08:00
CTL_PROTO(thread_idle)
Add the config.cache_oblivious mallctl.
2015-07-18 07:38:25 +08:00
CTL_PROTO(config_cache_oblivious)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(config_debug)
CTL_PROTO(config_fill)
CTL_PROTO(config_lazy_lock)
Add --with-malloc-conf. Add --with-malloc-conf, which makes it possible to embed a default options string during configuration.
2016-02-08 06:23:22 +08:00
CTL_PROTO(config_malloc_conf)
Safety checks: Expose config value via mallctl and stats.
2019-03-23 08:13:45 +08:00
CTL_PROTO(config_opt_safety_checks)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
CTL_PROTO(config_prof)
CTL_PROTO(config_prof_libgcc)
CTL_PROTO(config_prof_libunwind)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(config_stats)
Add utrace(2)-based tracing (--enable-utrace).
2012-04-06 04:36:17 +08:00
CTL_PROTO(config_utrace)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(config_xmalloc)
CTL_PROTO(opt_abort)
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
CTL_PROTO(opt_abort_conf)
Add confirm_conf option If the confirm_conf option is set, when the program starts, each of the four malloc_conf strings will be printed, and each option will be printed when being set.
2019-05-01 04:54:00 +08:00
CTL_PROTO(opt_confirm_conf)
Implement opt.metadata_thp This option enables transparent huge page for base allocators (require MADV_HUGEPAGE support).
2017-08-11 04:14:26 +08:00
CTL_PROTO(opt_metadata_thp)
Refactor !opt.munmap to opt.retain.
2017-04-27 07:26:12 +08:00
CTL_PROTO(opt_retain)
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
CTL_PROTO(opt_dss)
Replace JEMALLOC_OPTIONS with MALLOC_CONF. Replace the single-character run-time flags with key/value pairs, which can be set via the malloc_conf global, /etc/malloc.conf, and the MALLOC_CONF environment variable. Replace the JEMALLOC_PROF_PREFIX environment variable with the "opt.prof_prefix" option. Replace umax2s() with u2s().
2010-10-24 09:37:06 +08:00
CTL_PROTO(opt_narenas)
Implement per-CPU arena. The new feature, opt.percpu_arena, determines thread-arena association dynamically based CPU id. Three modes are supported: "percpu", "phycpu" and disabled. "percpu" uses the current core id (with help from sched_getcpu()) directly as the arena index, while "phycpu" will assign threads on the same physical CPU to the same arena. In other words, "percpu" means # of arenas == # of CPUs, while "phycpu" has # of arenas == 1/2 * (# of CPUs). Note that no runtime check on whether hyper threading is enabled is added yet. When enabled, threads will be migrated between arenas when a CPU change is detected. In the current design, to reduce overhead from reading CPU id, each arena tracks the thread accessed most recently. When a new thread comes in, we will read CPU id and update arena if necessary.
2017-02-03 09:02:05 +08:00
CTL_PROTO(opt_percpu_arena)
Rename huge_threshold to oversize_threshold. The keyword huge tend to remind people of huge pages which is not relevent to the feature.
2019-01-25 08:15:04 +08:00
CTL_PROTO(opt_oversize_threshold)
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
CTL_PROTO(opt_background_thread)
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
CTL_PROTO(opt_max_background_threads)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_PROTO(opt_dirty_decay_ms)
CTL_PROTO(opt_muzzy_decay_ms)
Replace JEMALLOC_OPTIONS with MALLOC_CONF. Replace the single-character run-time flags with key/value pairs, which can be set via the malloc_conf global, /etc/malloc.conf, and the MALLOC_CONF environment variable. Replace the JEMALLOC_PROF_PREFIX environment variable with the "opt.prof_prefix" option. Replace umax2s() with u2s().
2010-10-24 09:37:06 +08:00
CTL_PROTO(opt_stats_print)
Add opt.stats_print_opts. The value is passed to atexit(3)-triggered malloc_stats_print() calls.
2017-05-28 06:35:36 +08:00
CTL_PROTO(opt_stats_print_opts)
Implement opt.stats_interval and the _opts options. Add options stats_interval and stats_interval_opts to allow interval based stats printing. This provides an easy way to collect stats without code changes, because opt.stats_print may not work (some binaries never exit).
2020-01-14 14:29:17 +08:00
CTL_PROTO(opt_stats_interval)
CTL_PROTO(opt_stats_interval_opts)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(opt_junk)
Replace JEMALLOC_OPTIONS with MALLOC_CONF. Replace the single-character run-time flags with key/value pairs, which can be set via the malloc_conf global, /etc/malloc.conf, and the MALLOC_CONF environment variable. Replace the JEMALLOC_PROF_PREFIX environment variable with the "opt.prof_prefix" option. Replace umax2s() with u2s().
2010-10-24 09:37:06 +08:00
CTL_PROTO(opt_zero)
Add utrace(2)-based tracing (--enable-utrace).
2012-04-06 04:36:17 +08:00
CTL_PROTO(opt_utrace)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(opt_xmalloc)
Simplify tcache object caching. Use chains of cached objects, rather than using arrays of pointers. Since tcache_bin_t is no longer dynamically sized, convert tcache_t's tbin to an array of structures, rather than an array of pointers. This implicitly removes tcache_bin_{create,destroy}(), which further simplifies the fast path for malloc/free. Use cacheline alignment for tcache_t allocations. Remove runtime configuration option for number of tcache bin slots, and replace it with a boolean option for enabling/disabling tcache. Limit the number of tcache objects to the lesser of TCACHE_NSLOTS_MAX and 2X the number of regions per run for the size class. For GC-triggered flush, discard 3/4 of the objects below the low water mark, rather than 1/2.
2010-03-08 07:34:14 +08:00
CTL_PROTO(opt_tcache)
Add opt.thp which allows explicit hugepage usage. "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
2018-02-17 06:19:19 +08:00
CTL_PROTO(opt_thp)
Add opt.lg_extent_max_active_fit When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
2017-11-10 05:51:39 +08:00
CTL_PROTO(opt_lg_extent_max_active_fit)
Add missing "opt.lg_tcache_max" mallctl implementation.
2012-04-05 07:16:09 +08:00
CTL_PROTO(opt_lg_tcache_max)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
CTL_PROTO(opt_prof)
Replace JEMALLOC_OPTIONS with MALLOC_CONF. Replace the single-character run-time flags with key/value pairs, which can be set via the malloc_conf global, /etc/malloc.conf, and the MALLOC_CONF environment variable. Replace the JEMALLOC_PROF_PREFIX environment variable with the "opt.prof_prefix" option. Replace umax2s() with u2s().
2010-10-24 09:37:06 +08:00
CTL_PROTO(opt_prof_prefix)
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
CTL_PROTO(opt_prof_active)
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
CTL_PROTO(opt_prof_thread_active_init)
Implement sampling for heap profiling.
2010-03-02 12:15:26 +08:00
CTL_PROTO(opt_lg_prof_sample)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
CTL_PROTO(opt_lg_prof_interval)
Replace JEMALLOC_OPTIONS with MALLOC_CONF. Replace the single-character run-time flags with key/value pairs, which can be set via the malloc_conf global, /etc/malloc.conf, and the MALLOC_CONF environment variable. Replace the JEMALLOC_PROF_PREFIX environment variable with the "opt.prof_prefix" option. Replace umax2s() with u2s().
2010-10-24 09:37:06 +08:00
CTL_PROTO(opt_prof_gdump)
Update prof defaults to match common usage. Change the "opt.lg_prof_sample" default from 0 to 19 (1 B to 512 KiB). Change the "opt.prof_accum" default from true to false. Add the "opt.prof_final" mallctl, so that "opt.prof_prefix" need not be abused to disable final profile dumping.
2012-04-18 07:39:33 +08:00
CTL_PROTO(opt_prof_final)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
CTL_PROTO(opt_prof_leak)
Make cumulative heap profile data optional. Add the R option to control whether cumulative heap profile data are maintained. Add the T option to control the size of per thread backtrace caches, primarily because when the R option is specified, backtraces that no longer have allocations associated with them are discarded as soon as no thread caches refer to them.
2010-10-03 06:18:50 +08:00
CTL_PROTO(opt_prof_accum)
Last-N profiling mode
2019-12-19 05:38:14 +08:00
CTL_PROTO(opt_prof_recent_alloc_max)
Add option to fetch system thread name on each prof sample
2020-03-25 08:53:41 +08:00
CTL_PROTO(opt_prof_experimental_use_sys_thread_name)
Realloc: Make behavior of realloc(ptr, 0) configurable.
2019-09-24 08:56:19 +08:00
CTL_PROTO(opt_zero_realloc)
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
CTL_PROTO(tcache_create)
CTL_PROTO(tcache_flush)
CTL_PROTO(tcache_destroy)
Replace the arenas.initialized mallctl with arena.<i>.initialized .
2017-01-05 02:21:53 +08:00
CTL_PROTO(arena_i_initialized)
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
CTL_PROTO(arena_i_decay)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_PROTO(arena_i_purge)
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
CTL_PROTO(arena_i_reset)
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
CTL_PROTO(arena_i_destroy)
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
CTL_PROTO(arena_i_dss)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_PROTO(arena_i_dirty_decay_ms)
CTL_PROTO(arena_i_muzzy_decay_ms)
s/chunk_hook/extent_hook/g
2016-06-01 06:03:51 +08:00
CTL_PROTO(arena_i_extent_hooks)
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
CTL_PROTO(arena_i_retain_grow_limit)
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
INDEX_PROTO(arena_i)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(arenas_bin_i_size)
CTL_PROTO(arenas_bin_i_nregs)
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
CTL_PROTO(arenas_bin_i_slab_size)
Add stats for arenas.bin.i.nshards.
2018-11-22 03:17:31 +08:00
CTL_PROTO(arenas_bin_i_nshards)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
INDEX_PROTO(arenas_bin_i)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_PROTO(arenas_lextent_i_size)
INDEX_PROTO(arenas_lextent_i)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(arenas_narenas)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_PROTO(arenas_dirty_decay_ms)
CTL_PROTO(arenas_muzzy_decay_ms)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(arenas_quantum)
Clean up *PAGE* macros. s/PAGE_SHIFT/LG_PAGE/g and s/PAGE_SIZE/PAGE/g. Remove remnants of the dynamic-page-shift code. Rename the "arenas.pagesize" mallctl to "arenas.page". Remove the "arenas.chunksize" mallctl, which is redundant with "opt.lg_chunk".
2012-04-02 22:04:34 +08:00
CTL_PROTO(arenas_page)
Remove medium size classes. Remove medium size classes, because concurrent dirty page purging is no longer capable of purging inactive dirty pages inside active runs (due to recent arena/bin locking changes). Enhance tcache to support caching large objects, so that the same range of size classes is still cached, despite the removal of medium size class support.
2010-03-18 07:27:39 +08:00
CTL_PROTO(arenas_tcache_max)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(arenas_nbins)
Remove medium size classes. Remove medium size classes, because concurrent dirty page purging is no longer capable of purging inactive dirty pages inside active runs (due to recent arena/bin locking changes). Enhance tcache to support caching large objects, so that the same range of size classes is still cached, despite the removal of medium size class support.
2010-03-18 07:27:39 +08:00
CTL_PROTO(arenas_nhbins)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_PROTO(arenas_nlextents)
Rename the arenas.extend mallctl to arenas.create.
2017-01-04 00:21:29 +08:00
CTL_PROTO(arenas_create)
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
CTL_PROTO(arenas_lookup)
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
CTL_PROTO(prof_thread_active_init)
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
CTL_PROTO(prof_active)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
CTL_PROTO(prof_dump)
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
CTL_PROTO(prof_gdump)
Add "prof.dump_prefix" to override filename prefixes for dumps.
2019-09-10 11:18:41 +08:00
CTL_PROTO(prof_dump_prefix)
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
CTL_PROTO(prof_reset)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
CTL_PROTO(prof_interval)
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
CTL_PROTO(lg_prof_sample)
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
CTL_PROTO(prof_log_start)
CTL_PROTO(prof_log_stop)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(stats_arenas_i_small_allocated)
CTL_PROTO(stats_arenas_i_small_nmalloc)
CTL_PROTO(stats_arenas_i_small_ndalloc)
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
CTL_PROTO(stats_arenas_i_small_nrequests)
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
CTL_PROTO(stats_arenas_i_small_nfills)
CTL_PROTO(stats_arenas_i_small_nflushes)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_PROTO(stats_arenas_i_large_allocated)
CTL_PROTO(stats_arenas_i_large_nmalloc)
CTL_PROTO(stats_arenas_i_large_ndalloc)
CTL_PROTO(stats_arenas_i_large_nrequests)
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
CTL_PROTO(stats_arenas_i_large_nfills)
CTL_PROTO(stats_arenas_i_large_nflushes)
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
CTL_PROTO(stats_arenas_i_bins_j_nmalloc)
CTL_PROTO(stats_arenas_i_bins_j_ndalloc)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(stats_arenas_i_bins_j_nrequests)
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
CTL_PROTO(stats_arenas_i_bins_j_curregs)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(stats_arenas_i_bins_j_nfills)
CTL_PROTO(stats_arenas_i_bins_j_nflushes)
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
CTL_PROTO(stats_arenas_i_bins_j_nslabs)
CTL_PROTO(stats_arenas_i_bins_j_nreslabs)
CTL_PROTO(stats_arenas_i_bins_j_curslabs)
Add nonfull_slabs to bin_stats_t. When config_stats is enabled track the size of bin->slabs_nonfull in the new nonfull_slabs counter in bin_stats_t. This metric should be useful for establishing an upper ceiling on the savings possible by meshing.
2019-04-12 19:08:50 +08:00
CTL_PROTO(stats_arenas_i_bins_j_nonfull_slabs)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
INDEX_PROTO(stats_arenas_i_bins_j)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_PROTO(stats_arenas_i_lextents_j_nmalloc)
CTL_PROTO(stats_arenas_i_lextents_j_ndalloc)
CTL_PROTO(stats_arenas_i_lextents_j_nrequests)
CTL_PROTO(stats_arenas_i_lextents_j_curlextents)
INDEX_PROTO(stats_arenas_i_lextents_j)
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
CTL_PROTO(stats_arenas_i_extents_j_ndirty)
CTL_PROTO(stats_arenas_i_extents_j_nmuzzy)
CTL_PROTO(stats_arenas_i_extents_j_nretained)
CTL_PROTO(stats_arenas_i_extents_j_dirty_bytes)
CTL_PROTO(stats_arenas_i_extents_j_muzzy_bytes)
CTL_PROTO(stats_arenas_i_extents_j_retained_bytes)
INDEX_PROTO(stats_arenas_i_extents_j)
Improve thread-->arena assignment. Rather than blindly assigning threads to arenas in round-robin fashion, choose the lowest-numbered arena that currently has the smallest number of threads assigned to it. Add the "stats.arenas.<i>.nthreads" mallctl.
2011-03-19 04:41:33 +08:00
CTL_PROTO(stats_arenas_i_nthreads)
Add stats: arena uptime.
2017-05-17 04:56:00 +08:00
CTL_PROTO(stats_arenas_i_uptime)
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
CTL_PROTO(stats_arenas_i_dss)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_PROTO(stats_arenas_i_dirty_decay_ms)
CTL_PROTO(stats_arenas_i_muzzy_decay_ms)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(stats_arenas_i_pactive)
CTL_PROTO(stats_arenas_i_pdirty)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_PROTO(stats_arenas_i_pmuzzy)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(stats_arenas_i_mapped)
Add the stats.retained and stats.arenas.<i>.retained statistics. This resolves #367.
2016-05-04 13:11:35 +08:00
CTL_PROTO(stats_arenas_i_retained)
Add stats for the size of extent_avail heap
2018-08-02 05:14:33 +08:00
CTL_PROTO(stats_arenas_i_extent_avail)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_PROTO(stats_arenas_i_dirty_npurge)
CTL_PROTO(stats_arenas_i_dirty_nmadvise)
CTL_PROTO(stats_arenas_i_dirty_purged)
CTL_PROTO(stats_arenas_i_muzzy_npurge)
CTL_PROTO(stats_arenas_i_muzzy_nmadvise)
CTL_PROTO(stats_arenas_i_muzzy_purged)
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
CTL_PROTO(stats_arenas_i_base)
CTL_PROTO(stats_arenas_i_internal)
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
CTL_PROTO(stats_arenas_i_metadata_thp)
Added stats about number of bytes cached in tcache currently.
2016-04-23 09:37:44 +08:00
CTL_PROTO(stats_arenas_i_tcache_bytes)
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
CTL_PROTO(stats_arenas_i_resident)
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
CTL_PROTO(stats_arenas_i_abandoned_vm)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
INDEX_PROTO(stats_arenas_i)
CTL_PROTO(stats_allocated)
CTL_PROTO(stats_active)
Add background thread related stats.
2017-05-13 03:30:33 +08:00
CTL_PROTO(stats_background_thread_num_threads)
CTL_PROTO(stats_background_thread_num_runs)
CTL_PROTO(stats_background_thread_run_interval)
Implement metadata statistics. There are three categories of metadata: - Base allocations are used for bootstrap-sensitive internal allocator data structures. - Arena chunk headers comprise pages which track the states of the non-metadata pages. - Internal allocations differ from application-originated allocations in that they are for internal use, and that they are omitted from heap profiles. The metadata statistics comprise the metadata categories as follows: - stats.metadata: All metadata -- base + arena chunk headers + internal allocations. - stats.arenas.<i>.metadata.mapped: Arena chunk headers. - stats.arenas.<i>.metadata.allocated: Internal allocations. This is reported separately from the other metadata statistics because it overlaps with the allocated and active statistics, whereas the other metadata statistics do not. Base allocations are not reported separately, though their magnitude can be computed by subtracting the arena-specific metadata. This resolves #163.
2014-11-28 03:22:36 +08:00
CTL_PROTO(stats_metadata)
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
CTL_PROTO(stats_metadata_thp)
Add the "stats.allocated" mallctl.
2015-03-24 08:25:57 +08:00
CTL_PROTO(stats_resident)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
CTL_PROTO(stats_mapped)
Add the stats.retained and stats.arenas.<i>.retained statistics. This resolves #367.
2016-05-04 13:11:35 +08:00
CTL_PROTO(stats_retained)
Add stats counters for number of zero reallocs
2019-10-27 02:04:46 +08:00
CTL_PROTO(stats_zero_reallocs)
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
CTL_PROTO(experimental_hooks_install)
CTL_PROTO(experimental_hooks_remove)
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
CTL_PROTO(experimental_utilization_query)
CTL_PROTO(experimental_utilization_batch_query)
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
CTL_PROTO(experimental_arenas_i_pactivep)
INDEX_PROTO(experimental_arenas_i)
Last-N profiling mode
2019-12-19 05:38:14 +08:00
CTL_PROTO(experimental_prof_recent_alloc_max)
Add mallctl for dumping last-N profiling records
2020-02-05 08:05:11 +08:00
CTL_PROTO(experimental_prof_recent_alloc_dump)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
#define MUTEX_STATS_CTL_PROTO_GEN(n) \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL_PROTO(stats_##n##_num_ops) \
CTL_PROTO(stats_##n##_num_wait) \
CTL_PROTO(stats_##n##_num_spin_acq) \
CTL_PROTO(stats_##n##_num_owner_switch) \
CTL_PROTO(stats_##n##_total_wait_time) \
CTL_PROTO(stats_##n##_max_wait_time) \
CTL_PROTO(stats_##n##_max_num_thds)
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
/* Global mutexes. */
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) MUTEX_STATS_CTL_PROTO_GEN(mutexes_##mtx)
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_GLOBAL_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
/* Per arena mutexes. */
#define OP(mtx) MUTEX_STATS_CTL_PROTO_GEN(arenas_i_mutexes_##mtx)
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_ARENA_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
/* Arena bin mutexes. */
MUTEX_STATS_CTL_PROTO_GEN(arenas_i_bins_j_mutex)
#undef MUTEX_STATS_CTL_PROTO_GEN
CTL_PROTO(stats_mutexes_reset)
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/******************************************************************************/
/* mallctl tree. */
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NAME(n) {true}, n
#define CHILD(t, c) \
Fix ctl regression. Fix ctl to correctly compute the number of children at each level of the ctl tree.
2012-04-24 10:31:45 +08:00
sizeof(c##_node) / sizeof(ctl_##t##_node_t), \
(ctl_node_t *)c##_node, \
NULL
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL(c) 0, NULL, c##_ctl
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/*
* Only handles internal indexed nodes, since there are currently no external
* ones.
*/
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define INDEX(i) {false}, i##_index
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
static const ctl_named_node_t thread_tcache_node[] = {
Add the "thread.tcache.enabled" mallctl.
2012-03-27 09:54:44 +08:00
{NAME("enabled"), CTL(thread_tcache_enabled)},
Rename the "tcache.flush" mallctl to "thread.tcache.flush".
2012-03-17 08:09:32 +08:00
{NAME("flush"), CTL(thread_tcache_flush)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
static const ctl_named_node_t thread_prof_node[] = {
{NAME("name"), CTL(thread_prof_name)},
{NAME("active"), CTL(thread_prof_active)}
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t thread_node[] = {
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
{NAME("arena"), CTL(thread_arena)},
Add per thread allocation counters, and enhance heap sampling. Add the "thread.allocated" and "thread.deallocated" mallctls, which can be used to query the total number of bytes ever allocated/deallocated by the calling thread. Add s2u() and sa2u(), which can be used to compute the usable size that will result from an allocation request of a particular size/alignment. Re-factor ipalloc() to use sa2u(). Enhance the heap profiler to trigger samples based on usable size, rather than request size. This has a subtle, but important, impact on the accuracy of heap sampling. For example, previous to this change, 16- and 17-byte objects were sampled at nearly the same rate, but 17-byte objects actually consume 32 bytes each. Therefore it was possible for the sample to be somewhat skewed compared to actual memory usage of the allocated objects.
2010-10-21 08:39:18 +08:00
{NAME("allocated"), CTL(thread_allocated)},
Add the "thread.[de]allocatedp" mallctl's.
2010-12-04 07:55:47 +08:00
{NAME("allocatedp"), CTL(thread_allocatedp)},
{NAME("deallocated"), CTL(thread_deallocated)},
Rename the "tcache.flush" mallctl to "thread.tcache.flush".
2012-03-17 08:09:32 +08:00
{NAME("deallocatedp"), CTL(thread_deallocatedp)},
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
{NAME("tcache"), CHILD(named, thread_tcache)},
Add "thread.idle" mallctl. This can encapsulate various internal cleaning logic, and can be used to free up resources before a long sleep.
2020-01-23 03:13:26 +08:00
{NAME("prof"), CHILD(named, thread_prof)},
{NAME("idle"), CTL(thread_idle)}
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t config_node[] = {
Add the config.cache_oblivious mallctl.
2015-07-18 07:38:25 +08:00
{NAME("cache_oblivious"), CTL(config_cache_oblivious)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("debug"), CTL(config_debug)},
{NAME("fill"), CTL(config_fill)},
{NAME("lazy_lock"), CTL(config_lazy_lock)},
Add --with-malloc-conf. Add --with-malloc-conf, which makes it possible to embed a default options string during configuration.
2016-02-08 06:23:22 +08:00
{NAME("malloc_conf"), CTL(config_malloc_conf)},
Safety checks: Expose config value via mallctl and stats.
2019-03-23 08:13:45 +08:00
{NAME("opt_safety_checks"), CTL(config_opt_safety_checks)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("prof"), CTL(config_prof)},
{NAME("prof_libgcc"), CTL(config_prof_libgcc)},
{NAME("prof_libunwind"), CTL(config_prof_libunwind)},
{NAME("stats"), CTL(config_stats)},
{NAME("utrace"), CTL(config_utrace)},
{NAME("xmalloc"), CTL(config_xmalloc)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t opt_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("abort"), CTL(opt_abort)},
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
{NAME("abort_conf"), CTL(opt_abort_conf)},
Add confirm_conf option If the confirm_conf option is set, when the program starts, each of the four malloc_conf strings will be printed, and each option will be printed when being set.
2019-05-01 04:54:00 +08:00
{NAME("confirm_conf"), CTL(opt_confirm_conf)},
Implement opt.metadata_thp This option enables transparent huge page for base allocators (require MADV_HUGEPAGE support).
2017-08-11 04:14:26 +08:00
{NAME("metadata_thp"), CTL(opt_metadata_thp)},
Refactor !opt.munmap to opt.retain.
2017-04-27 07:26:12 +08:00
{NAME("retain"), CTL(opt_retain)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("dss"), CTL(opt_dss)},
{NAME("narenas"), CTL(opt_narenas)},
Implement per-CPU arena. The new feature, opt.percpu_arena, determines thread-arena association dynamically based CPU id. Three modes are supported: "percpu", "phycpu" and disabled. "percpu" uses the current core id (with help from sched_getcpu()) directly as the arena index, while "phycpu" will assign threads on the same physical CPU to the same arena. In other words, "percpu" means # of arenas == # of CPUs, while "phycpu" has # of arenas == 1/2 * (# of CPUs). Note that no runtime check on whether hyper threading is enabled is added yet. When enabled, threads will be migrated between arenas when a CPU change is detected. In the current design, to reduce overhead from reading CPU id, each arena tracks the thread accessed most recently. When a new thread comes in, we will read CPU id and update arena if necessary.
2017-02-03 09:02:05 +08:00
{NAME("percpu_arena"), CTL(opt_percpu_arena)},
Rename huge_threshold to oversize_threshold. The keyword huge tend to remind people of huge pages which is not relevent to the feature.
2019-01-25 08:15:04 +08:00
{NAME("oversize_threshold"), CTL(opt_oversize_threshold)},
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
{NAME("background_thread"), CTL(opt_background_thread)},
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
{NAME("max_background_threads"), CTL(opt_max_background_threads)},
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
{NAME("dirty_decay_ms"), CTL(opt_dirty_decay_ms)},
{NAME("muzzy_decay_ms"), CTL(opt_muzzy_decay_ms)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("stats_print"), CTL(opt_stats_print)},
Add opt.stats_print_opts. The value is passed to atexit(3)-triggered malloc_stats_print() calls.
2017-05-28 06:35:36 +08:00
{NAME("stats_print_opts"), CTL(opt_stats_print_opts)},
Implement opt.stats_interval and the _opts options. Add options stats_interval and stats_interval_opts to allow interval based stats printing. This provides an easy way to collect stats without code changes, because opt.stats_print may not work (some binaries never exit).
2020-01-14 14:29:17 +08:00
{NAME("stats_interval"), CTL(opt_stats_interval)},
{NAME("stats_interval_opts"), CTL(opt_stats_interval_opts)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("junk"), CTL(opt_junk)},
{NAME("zero"), CTL(opt_zero)},
{NAME("utrace"), CTL(opt_utrace)},
{NAME("xmalloc"), CTL(opt_xmalloc)},
{NAME("tcache"), CTL(opt_tcache)},
Add opt.thp which allows explicit hugepage usage. "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
2018-02-17 06:19:19 +08:00
{NAME("thp"), CTL(opt_thp)},
Add opt.lg_extent_max_active_fit When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
2017-11-10 05:51:39 +08:00
{NAME("lg_extent_max_active_fit"), CTL(opt_lg_extent_max_active_fit)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("lg_tcache_max"), CTL(opt_lg_tcache_max)},
{NAME("prof"), CTL(opt_prof)},
{NAME("prof_prefix"), CTL(opt_prof_prefix)},
{NAME("prof_active"), CTL(opt_prof_active)},
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
{NAME("prof_thread_active_init"), CTL(opt_prof_thread_active_init)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("lg_prof_sample"), CTL(opt_lg_prof_sample)},
{NAME("lg_prof_interval"), CTL(opt_lg_prof_interval)},
{NAME("prof_gdump"), CTL(opt_prof_gdump)},
{NAME("prof_final"), CTL(opt_prof_final)},
{NAME("prof_leak"), CTL(opt_prof_leak)},
Realloc: Make behavior of realloc(ptr, 0) configurable.
2019-09-24 08:56:19 +08:00
{NAME("prof_accum"), CTL(opt_prof_accum)},
Last-N profiling mode
2019-12-19 05:38:14 +08:00
{NAME("prof_recent_alloc_max"), CTL(opt_prof_recent_alloc_max)},
Add option to fetch system thread name on each prof sample
2020-03-25 08:53:41 +08:00
{NAME("prof_experimental_use_sys_thread_name"),
CTL(opt_prof_experimental_use_sys_thread_name)},
Realloc: Make behavior of realloc(ptr, 0) configurable.
2019-09-24 08:56:19 +08:00
{NAME("zero_realloc"), CTL(opt_zero_realloc)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
static const ctl_named_node_t tcache_node[] = {
{NAME("create"), CTL(tcache_create)},
{NAME("flush"), CTL(tcache_flush)},
{NAME("destroy"), CTL(tcache_destroy)}
};
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static const ctl_named_node_t arena_i_node[] = {
Replace the arenas.initialized mallctl with arena.<i>.initialized .
2017-01-05 02:21:53 +08:00
{NAME("initialized"), CTL(arena_i_initialized)},
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
{NAME("decay"), CTL(arena_i_decay)},
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
{NAME("purge"), CTL(arena_i_purge)},
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
{NAME("reset"), CTL(arena_i_reset)},
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
{NAME("destroy"), CTL(arena_i_destroy)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("dss"), CTL(arena_i_dss)},
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
{NAME("dirty_decay_ms"), CTL(arena_i_dirty_decay_ms)},
{NAME("muzzy_decay_ms"), CTL(arena_i_muzzy_decay_ms)},
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
{NAME("extent_hooks"), CTL(arena_i_extent_hooks)},
{NAME("retain_grow_limit"), CTL(arena_i_retain_grow_limit)}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
};
static const ctl_named_node_t super_arena_i_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME(""), CHILD(named, arena_i)}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
};
static const ctl_indexed_node_t arena_node[] = {
{INDEX(arena_i)}
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t arenas_bin_i_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("size"), CTL(arenas_bin_i_size)},
{NAME("nregs"), CTL(arenas_bin_i_nregs)},
Add stats for arenas.bin.i.nshards.
2018-11-22 03:17:31 +08:00
{NAME("slab_size"), CTL(arenas_bin_i_slab_size)},
{NAME("nshards"), CTL(arenas_bin_i_nshards)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t super_arenas_bin_i_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME(""), CHILD(named, arenas_bin_i)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_indexed_node_t arenas_bin_node[] = {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
{INDEX(arenas_bin_i)}
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_named_node_t arenas_lextent_i_node[] = {
{NAME("size"), CTL(arenas_lextent_i_size)}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_named_node_t super_arenas_lextent_i_node[] = {
{NAME(""), CHILD(named, arenas_lextent_i)}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_indexed_node_t arenas_lextent_node[] = {
{INDEX(arenas_lextent_i)}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t arenas_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("narenas"), CTL(arenas_narenas)},
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
{NAME("dirty_decay_ms"), CTL(arenas_dirty_decay_ms)},
{NAME("muzzy_decay_ms"), CTL(arenas_muzzy_decay_ms)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("quantum"), CTL(arenas_quantum)},
{NAME("page"), CTL(arenas_page)},
{NAME("tcache_max"), CTL(arenas_tcache_max)},
{NAME("nbins"), CTL(arenas_nbins)},
{NAME("nhbins"), CTL(arenas_nhbins)},
{NAME("bin"), CHILD(indexed, arenas_bin)},
Rename huge to large.
2016-06-01 05:50:21 +08:00
{NAME("nlextents"), CTL(arenas_nlextents)},
{NAME("lextent"), CHILD(indexed, arenas_lextent)},
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
{NAME("create"), CTL(arenas_create)},
{NAME("lookup"), CTL(arenas_lookup)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t prof_node[] = {
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
{NAME("thread_active_init"), CTL(prof_thread_active_init)},
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
{NAME("active"), CTL(prof_active)},
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
{NAME("dump"), CTL(prof_dump)},
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
{NAME("gdump"), CTL(prof_gdump)},
Add "prof.dump_prefix" to override filename prefixes for dumps.
2019-09-10 11:18:41 +08:00
{NAME("dump_prefix"), CTL(prof_dump_prefix)},
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
{NAME("reset"), CTL(prof_reset)},
{NAME("interval"), CTL(prof_interval)},
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
{NAME("lg_sample"), CTL(lg_prof_sample)},
{NAME("log_start"), CTL(prof_log_start)},
{NAME("log_stop"), CTL(prof_log_stop)}
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t stats_arenas_i_small_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("allocated"), CTL(stats_arenas_i_small_allocated)},
{NAME("nmalloc"), CTL(stats_arenas_i_small_nmalloc)},
{NAME("ndalloc"), CTL(stats_arenas_i_small_ndalloc)},
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
{NAME("nrequests"), CTL(stats_arenas_i_small_nrequests)},
{NAME("nfills"), CTL(stats_arenas_i_small_nfills)},
{NAME("nflushes"), CTL(stats_arenas_i_small_nflushes)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_named_node_t stats_arenas_i_large_node[] = {
{NAME("allocated"), CTL(stats_arenas_i_large_allocated)},
{NAME("nmalloc"), CTL(stats_arenas_i_large_nmalloc)},
{NAME("ndalloc"), CTL(stats_arenas_i_large_ndalloc)},
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
{NAME("nrequests"), CTL(stats_arenas_i_large_nrequests)},
{NAME("nfills"), CTL(stats_arenas_i_large_nfills)},
{NAME("nflushes"), CTL(stats_arenas_i_large_nflushes)}
Refactor huge allocation to be managed by arenas. Refactor huge allocation to be managed by arenas (though the global red-black tree of huge allocations remains for lookup during deallocation). This is the logical conclusion of recent changes that 1) made per arena dss precedence apply to huge allocation, and 2) made it possible to replace the per arena chunk allocation/deallocation functions. Remove the top level huge stats, and replace them with per arena huge stats. Normalize function names and types to *dalloc* (some were *dealloc*). Remove the --enable-mremap option. As jemalloc currently operates, this is a performace regression for some applications, but planned work to logarithmically space huge size classes should provide similar amortized performance. The motivation for this change was that mremap-based huge reallocation forced leaky abstractions that prevented refactoring.
2014-05-16 13:22:27 +08:00
};
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
#define MUTEX_PROF_DATA_NODE(prefix) \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
static const ctl_named_node_t stats_##prefix##_node[] = { \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("num_ops"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_num_ops)}, \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("num_wait"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_num_wait)}, \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("num_spin_acq"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_num_spin_acq)}, \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("num_owner_switch"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_num_owner_switch)}, \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("total_wait_time"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_total_wait_time)}, \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("max_wait_time"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_max_wait_time)}, \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("max_num_thds"), \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL(stats_##prefix##_max_num_thds)} \
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
/* Note that # of current waiting thread not provided. */ \
};
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
MUTEX_PROF_DATA_NODE(arenas_i_bins_j_mutex)
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t stats_arenas_i_bins_j_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("nmalloc"), CTL(stats_arenas_i_bins_j_nmalloc)},
{NAME("ndalloc"), CTL(stats_arenas_i_bins_j_ndalloc)},
{NAME("nrequests"), CTL(stats_arenas_i_bins_j_nrequests)},
{NAME("curregs"), CTL(stats_arenas_i_bins_j_curregs)},
{NAME("nfills"), CTL(stats_arenas_i_bins_j_nfills)},
{NAME("nflushes"), CTL(stats_arenas_i_bins_j_nflushes)},
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
{NAME("nslabs"), CTL(stats_arenas_i_bins_j_nslabs)},
{NAME("nreslabs"), CTL(stats_arenas_i_bins_j_nreslabs)},
Output bin lock profiling results to malloc_stats. Two counters are included for the small bins: lock contention rate, and max lock waiting time.
2017-03-04 11:58:43 +08:00
{NAME("curslabs"), CTL(stats_arenas_i_bins_j_curslabs)},
Add nonfull_slabs to bin_stats_t. When config_stats is enabled track the size of bin->slabs_nonfull in the new nonfull_slabs counter in bin_stats_t. This metric should be useful for establishing an upper ceiling on the savings possible by meshing.
2019-04-12 19:08:50 +08:00
{NAME("nonfull_slabs"), CTL(stats_arenas_i_bins_j_nonfull_slabs)},
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
{NAME("mutex"), CHILD(named, stats_arenas_i_bins_j_mutex)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t super_stats_arenas_i_bins_j_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME(""), CHILD(named, stats_arenas_i_bins_j)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_indexed_node_t stats_arenas_i_bins_node[] = {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
{INDEX(stats_arenas_i_bins_j)}
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_named_node_t stats_arenas_i_lextents_j_node[] = {
{NAME("nmalloc"), CTL(stats_arenas_i_lextents_j_nmalloc)},
{NAME("ndalloc"), CTL(stats_arenas_i_lextents_j_ndalloc)},
{NAME("nrequests"), CTL(stats_arenas_i_lextents_j_nrequests)},
{NAME("curlextents"), CTL(stats_arenas_i_lextents_j_curlextents)}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_named_node_t super_stats_arenas_i_lextents_j_node[] = {
{NAME(""), CHILD(named, stats_arenas_i_lextents_j)}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
};
Rename huge to large.
2016-06-01 05:50:21 +08:00
static const ctl_indexed_node_t stats_arenas_i_lextents_node[] = {
{INDEX(stats_arenas_i_lextents_j)}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
};
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
static const ctl_named_node_t stats_arenas_i_extents_j_node[] = {
{NAME("ndirty"), CTL(stats_arenas_i_extents_j_ndirty)},
{NAME("nmuzzy"), CTL(stats_arenas_i_extents_j_nmuzzy)},
{NAME("nretained"), CTL(stats_arenas_i_extents_j_nretained)},
{NAME("dirty_bytes"), CTL(stats_arenas_i_extents_j_dirty_bytes)},
{NAME("muzzy_bytes"), CTL(stats_arenas_i_extents_j_muzzy_bytes)},
{NAME("retained_bytes"), CTL(stats_arenas_i_extents_j_retained_bytes)}
};
static const ctl_named_node_t super_stats_arenas_i_extents_j_node[] = {
{NAME(""), CHILD(named, stats_arenas_i_extents_j)}
};
static const ctl_indexed_node_t stats_arenas_i_extents_node[] = {
{INDEX(stats_arenas_i_extents_j)}
};
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) MUTEX_PROF_DATA_NODE(arenas_i_mutexes_##mtx)
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_ARENA_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
static const ctl_named_node_t stats_arenas_i_mutexes_node[] = {
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) {NAME(#mtx), CHILD(named, stats_arenas_i_mutexes_##mtx)},
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_ARENA_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t stats_arenas_i_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("nthreads"), CTL(stats_arenas_i_nthreads)},
Add stats: arena uptime.
2017-05-17 04:56:00 +08:00
{NAME("uptime"), CTL(stats_arenas_i_uptime)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("dss"), CTL(stats_arenas_i_dss)},
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
{NAME("dirty_decay_ms"), CTL(stats_arenas_i_dirty_decay_ms)},
{NAME("muzzy_decay_ms"), CTL(stats_arenas_i_muzzy_decay_ms)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("pactive"), CTL(stats_arenas_i_pactive)},
{NAME("pdirty"), CTL(stats_arenas_i_pdirty)},
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
{NAME("pmuzzy"), CTL(stats_arenas_i_pmuzzy)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("mapped"), CTL(stats_arenas_i_mapped)},
Add the stats.retained and stats.arenas.<i>.retained statistics. This resolves #367.
2016-05-04 13:11:35 +08:00
{NAME("retained"), CTL(stats_arenas_i_retained)},
Add stats for the size of extent_avail heap
2018-08-02 05:14:33 +08:00
{NAME("extent_avail"), CTL(stats_arenas_i_extent_avail)},
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
{NAME("dirty_npurge"), CTL(stats_arenas_i_dirty_npurge)},
{NAME("dirty_nmadvise"), CTL(stats_arenas_i_dirty_nmadvise)},
{NAME("dirty_purged"), CTL(stats_arenas_i_dirty_purged)},
{NAME("muzzy_npurge"), CTL(stats_arenas_i_muzzy_npurge)},
{NAME("muzzy_nmadvise"), CTL(stats_arenas_i_muzzy_nmadvise)},
{NAME("muzzy_purged"), CTL(stats_arenas_i_muzzy_purged)},
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
{NAME("base"), CTL(stats_arenas_i_base)},
{NAME("internal"), CTL(stats_arenas_i_internal)},
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
{NAME("metadata_thp"), CTL(stats_arenas_i_metadata_thp)},
Added stats about number of bytes cached in tcache currently.
2016-04-23 09:37:44 +08:00
{NAME("tcache_bytes"), CTL(stats_arenas_i_tcache_bytes)},
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
{NAME("resident"), CTL(stats_arenas_i_resident)},
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
{NAME("abandoned_vm"), CTL(stats_arenas_i_abandoned_vm)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("small"), CHILD(named, stats_arenas_i_small)},
Rename huge to large.
2016-06-01 05:50:21 +08:00
{NAME("large"), CHILD(named, stats_arenas_i_large)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("bins"), CHILD(indexed, stats_arenas_i_bins)},
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
{NAME("lextents"), CHILD(indexed, stats_arenas_i_lextents)},
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
{NAME("extents"), CHILD(indexed, stats_arenas_i_extents)},
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
{NAME("mutexes"), CHILD(named, stats_arenas_i_mutexes)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t super_stats_arenas_i_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME(""), CHILD(named, stats_arenas_i)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_indexed_node_t stats_arenas_node[] = {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
{INDEX(stats_arenas_i)}
};
Add background thread related stats.
2017-05-13 03:30:33 +08:00
static const ctl_named_node_t stats_background_thread_node[] = {
{NAME("num_threads"), CTL(stats_background_thread_num_threads)},
{NAME("num_runs"), CTL(stats_background_thread_num_runs)},
{NAME("run_interval"), CTL(stats_background_thread_run_interval)}
};
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) MUTEX_PROF_DATA_NODE(mutexes_##mtx)
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_GLOBAL_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
static const ctl_named_node_t stats_mutexes_node[] = {
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) {NAME(#mtx), CHILD(named, stats_mutexes_##mtx)},
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_GLOBAL_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
{NAME("reset"), CTL(stats_mutexes_reset)}
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
};
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef MUTEX_PROF_DATA_NODE
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t stats_node[] = {
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("allocated"), CTL(stats_allocated)},
{NAME("active"), CTL(stats_active)},
Implement metadata statistics. There are three categories of metadata: - Base allocations are used for bootstrap-sensitive internal allocator data structures. - Arena chunk headers comprise pages which track the states of the non-metadata pages. - Internal allocations differ from application-originated allocations in that they are for internal use, and that they are omitted from heap profiles. The metadata statistics comprise the metadata categories as follows: - stats.metadata: All metadata -- base + arena chunk headers + internal allocations. - stats.arenas.<i>.metadata.mapped: Arena chunk headers. - stats.arenas.<i>.metadata.allocated: Internal allocations. This is reported separately from the other metadata statistics because it overlaps with the allocated and active statistics, whereas the other metadata statistics do not. Base allocations are not reported separately, though their magnitude can be computed by subtracting the arena-specific metadata. This resolves #163.
2014-11-28 03:22:36 +08:00
{NAME("metadata"), CTL(stats_metadata)},
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
{NAME("metadata_thp"), CTL(stats_metadata_thp)},
Add the "stats.allocated" mallctl.
2015-03-24 08:25:57 +08:00
{NAME("resident"), CTL(stats_resident)},
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{NAME("mapped"), CTL(stats_mapped)},
Add the stats.retained and stats.arenas.<i>.retained statistics. This resolves #367.
2016-05-04 13:11:35 +08:00
{NAME("retained"), CTL(stats_retained)},
Add background thread related stats.
2017-05-13 03:30:33 +08:00
{NAME("background_thread"),
CHILD(named, stats_background_thread)},
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
{NAME("mutexes"), CHILD(named, stats_mutexes)},
Add stats counters for number of zero reallocs
2019-10-27 02:04:46 +08:00
{NAME("arenas"), CHILD(indexed, stats_arenas)},
{NAME("zero_reallocs"), CTL(stats_zero_reallocs)},
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
static const ctl_named_node_t experimental_hooks_node[] = {
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
{NAME("install"), CTL(experimental_hooks_install)},
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
{NAME("remove"), CTL(experimental_hooks_remove)}
};
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
static const ctl_named_node_t experimental_utilization_node[] = {
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
{NAME("query"), CTL(experimental_utilization_query)},
{NAME("batch_query"), CTL(experimental_utilization_batch_query)}
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
};
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
static const ctl_named_node_t experimental_arenas_i_node[] = {
{NAME("pactivep"), CTL(experimental_arenas_i_pactivep)}
};
static const ctl_named_node_t super_experimental_arenas_i_node[] = {
{NAME(""), CHILD(named, experimental_arenas_i)}
};
static const ctl_indexed_node_t experimental_arenas_node[] = {
{INDEX(experimental_arenas_i)}
};
Last-N profiling mode
2019-12-19 05:38:14 +08:00
static const ctl_named_node_t experimental_prof_recent_node[] = {
{NAME("alloc_max"), CTL(experimental_prof_recent_alloc_max)},
Add mallctl for dumping last-N profiling records
2020-02-05 08:05:11 +08:00
{NAME("alloc_dump"), CTL(experimental_prof_recent_alloc_dump)},
Last-N profiling mode
2019-12-19 05:38:14 +08:00
};
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
static const ctl_named_node_t experimental_node[] = {
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
{NAME("hooks"), CHILD(named, experimental_hooks)},
{NAME("utilization"), CHILD(named, experimental_utilization)},
Last-N profiling mode
2019-12-19 05:38:14 +08:00
{NAME("arenas"), CHILD(indexed, experimental_arenas)},
{NAME("prof_recent"), CHILD(named, experimental_prof_recent)}
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t root_node[] = {
Add release versioning support. Base version string on 'git describe --long', and provide cpp macros in jemalloc.h. Add the version mallctl.
2010-03-03 05:01:16 +08:00
{NAME("version"), CTL(version)},
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
{NAME("epoch"), CTL(epoch)},
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
{NAME("background_thread"), CTL(background_thread)},
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
{NAME("max_background_threads"), CTL(max_background_threads)},
Fix ctl regression. Fix ctl to correctly compute the number of children at each level of the ctl tree.
2012-04-24 10:31:45 +08:00
{NAME("thread"), CHILD(named, thread)},
{NAME("config"), CHILD(named, config)},
{NAME("opt"), CHILD(named, opt)},
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
{NAME("tcache"), CHILD(named, tcache)},
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
{NAME("arena"), CHILD(indexed, arena)},
Fix ctl regression. Fix ctl to correctly compute the number of children at each level of the ctl tree.
2012-04-24 10:31:45 +08:00
{NAME("arenas"), CHILD(named, arenas)},
{NAME("prof"), CHILD(named, prof)},
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
{NAME("stats"), CHILD(named, stats)},
{NAME("experimental"), CHILD(named, experimental)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
static const ctl_named_node_t super_root_node[] = {
Fix ctl regression. Fix ctl to correctly compute the number of children at each level of the ctl tree.
2012-04-24 10:31:45 +08:00
{NAME(""), CHILD(named, root)}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
};
#undef NAME
#undef CHILD
#undef CTL
#undef INDEX
/******************************************************************************/
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
/*
* Sets *dst + *src non-atomically. This is safe, since everything is
* synchronized by the ctl mutex.
*/
static void
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(locked_u64_t *dst, locked_u64_t *src) {
locked_inc_u64_unsynchronized(dst,
locked_read_u64_unsynchronized(src));
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
}
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
static void
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_zu(locked_zu_t *dst, locked_zu_t *src) {
locked_inc_zu_unsynchronized(dst,
locked_read_atomic_zu(src));
}
static void
ctl_accum_atomic_zu(atomic_zu_t *dst, atomic_zu_t *src) {
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
size_t cur_dst = atomic_load_zu(dst, ATOMIC_RELAXED);
size_t cur_src = atomic_load_zu(src, ATOMIC_RELAXED);
atomic_store_zu(dst, cur_dst + cur_src, ATOMIC_RELAXED);
}
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
/******************************************************************************/
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
static unsigned
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arenas_i2a_impl(size_t i, bool compat, bool validate) {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
unsigned a;
switch (i) {
case MALLCTL_ARENAS_ALL:
a = 0;
break;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
case MALLCTL_ARENAS_DESTROYED:
a = 1;
break;
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
default:
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if (compat && i == ctl_arenas->narenas) {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
/*
* Provide deprecated backward compatibility for
* accessing the merged stats at index narenas rather
* than via MALLCTL_ARENAS_ALL. This is scheduled for
* removal in 6.0.0.
*/
a = 0;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else if (validate && i >= ctl_arenas->narenas) {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
a = UINT_MAX;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
/*
* This function should never be called for an index
* more than one past the range of indices that have
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
* initialized ctl data.
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
*/
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
assert(i < ctl_arenas->narenas || (!validate && i ==
ctl_arenas->narenas));
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
a = (unsigned)i + 2;
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
}
break;
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return a;
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
static unsigned
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arenas_i2a(size_t i) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return arenas_i2a_impl(i, true, false);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
static ctl_arena_t *
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
arenas_i_impl(tsd_t *tsd, size_t i, bool compat, bool init) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_t *ret;
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
assert(!compat || !init);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ret = ctl_arenas->arenas[arenas_i2a_impl(i, compat, false)];
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
if (init && ret == NULL) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if (config_stats) {
struct container_s {
ctl_arena_t ctl_arena;
ctl_arena_stats_t astats;
};
struct container_s *cont =
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
(struct container_s *)base_alloc(tsd_tsdn(tsd),
b0get(), sizeof(struct container_s), QUANTUM);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if (cont == NULL) {
return NULL;
}
ret = &cont->ctl_arena;
ret->astats = &cont->astats;
} else {
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
ret = (ctl_arena_t *)base_alloc(tsd_tsdn(tsd), b0get(),
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
sizeof(ctl_arena_t), QUANTUM);
if (ret == NULL) {
return NULL;
}
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
ret->arena_ind = (unsigned)i;
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arenas->arenas[arenas_i2a_impl(i, compat, false)] = ret;
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
assert(ret == NULL || arenas_i2a(ret->arena_ind) == arenas_i2a(i));
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
}
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
static ctl_arena_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arenas_i(size_t i) {
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
ctl_arena_t *ret = arenas_i_impl(tsd_fetch(), i, true, false);
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
assert(ret != NULL);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_arena_clear(ctl_arena_t *ctl_arena) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena->nthreads = 0;
ctl_arena->dss = dss_prec_names[dss_prec_limit];
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
ctl_arena->dirty_decay_ms = -1;
ctl_arena->muzzy_decay_ms = -1;
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena->pactive = 0;
ctl_arena->pdirty = 0;
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
ctl_arena->pmuzzy = 0;
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
if (config_stats) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
memset(&ctl_arena->astats->astats, 0, sizeof(arena_stats_t));
ctl_arena->astats->allocated_small = 0;
ctl_arena->astats->nmalloc_small = 0;
ctl_arena->astats->ndalloc_small = 0;
ctl_arena->astats->nrequests_small = 0;
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
ctl_arena->astats->nfills_small = 0;
ctl_arena->astats->nflushes_small = 0;
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
memset(ctl_arena->astats->bstats, 0, SC_NBINS *
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
sizeof(bin_stats_data_t));
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
memset(ctl_arena->astats->lstats, 0, (SC_NSIZES - SC_NBINS) *
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
sizeof(arena_stats_large_t));
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
memset(ctl_arena->astats->estats, 0, SC_NPSIZES *
sizeof(arena_stats_extents_t));
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_arena_stats_amerge(tsdn_t *tsdn, ctl_arena_t *ctl_arena, arena_t *arena) {
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
unsigned i;
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
if (config_stats) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arena_stats_merge(tsdn, arena, &ctl_arena->nthreads,
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
&ctl_arena->dss, &ctl_arena->dirty_decay_ms,
&ctl_arena->muzzy_decay_ms, &ctl_arena->pactive,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
&ctl_arena->pdirty, &ctl_arena->pmuzzy,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
&ctl_arena->astats->astats, ctl_arena->astats->bstats,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
ctl_arena->astats->lstats, ctl_arena->astats->estats);
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
for (i = 0; i < SC_NBINS; i++) {
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
bin_stats_t *bstats =
&ctl_arena->astats->bstats[i].stats_data;
ctl_arena->astats->allocated_small += bstats->curregs *
Header refactoring: Pull size helpers out of jemalloc module.
2017-05-31 01:45:37 +08:00
sz_index2size(i);
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
ctl_arena->astats->nmalloc_small += bstats->nmalloc;
ctl_arena->astats->ndalloc_small += bstats->ndalloc;
ctl_arena->astats->nrequests_small += bstats->nrequests;
ctl_arena->astats->nfills_small += bstats->nfills;
ctl_arena->astats->nflushes_small += bstats->nflushes;
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
}
} else {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arena_basic_stats_merge(tsdn, arena, &ctl_arena->nthreads,
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
&ctl_arena->dss, &ctl_arena->dirty_decay_ms,
&ctl_arena->muzzy_decay_ms, &ctl_arena->pactive,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
&ctl_arena->pdirty, &ctl_arena->pmuzzy);
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
}
}
static void
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_stats_sdmerge(ctl_arena_t *ctl_sdarena, ctl_arena_t *ctl_arena,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
bool destroyed) {
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
unsigned i;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_sdarena->nthreads += ctl_arena->nthreads;
ctl_sdarena->pactive += ctl_arena->pactive;
ctl_sdarena->pdirty += ctl_arena->pdirty;
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
ctl_sdarena->pmuzzy += ctl_arena->pmuzzy;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
} else {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
assert(ctl_arena->nthreads == 0);
assert(ctl_arena->pactive == 0);
assert(ctl_arena->pdirty == 0);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
assert(ctl_arena->pmuzzy == 0);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
}
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
if (config_stats) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_stats_t *sdstats = ctl_sdarena->astats;
ctl_arena_stats_t *astats = ctl_arena->astats;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
PA: Move mapped into pa stats.
2020-03-10 03:06:19 +08:00
ctl_accum_locked_zu(
&sdstats->astats.pa_shard_stats.mapped,
&astats->astats.pa_shard_stats.mapped);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_zu(&sdstats->astats.retained,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
&astats->astats.retained);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.edata_avail,
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
&astats->astats.edata_avail);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
}
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
ctl_accum_locked_u64(
&sdstats->astats.pa_shard_stats.decay_dirty.npurge,
&astats->astats.pa_shard_stats.decay_dirty.npurge);
ctl_accum_locked_u64(
&sdstats->astats.pa_shard_stats.decay_dirty.nmadvise,
&astats->astats.pa_shard_stats.decay_dirty.nmadvise);
ctl_accum_locked_u64(
&sdstats->astats.pa_shard_stats.decay_dirty.purged,
&astats->astats.pa_shard_stats.decay_dirty.purged);
ctl_accum_locked_u64(
&sdstats->astats.pa_shard_stats.decay_muzzy.npurge,
&astats->astats.pa_shard_stats.decay_muzzy.npurge);
ctl_accum_locked_u64(
&sdstats->astats.pa_shard_stats.decay_muzzy.nmadvise,
&astats->astats.pa_shard_stats.decay_muzzy.nmadvise);
ctl_accum_locked_u64(
&sdstats->astats.pa_shard_stats.decay_muzzy.purged,
&astats->astats.pa_shard_stats.decay_muzzy.purged);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) malloc_mutex_prof_merge( \
&(sdstats->astats.mutex_prof_data[ \
arena_prof_mutex_##mtx]), \
&(astats->astats.mutex_prof_data[ \
arena_prof_mutex_##mtx]));
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_ARENA_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.base,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
&astats->astats.base);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.internal,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
&astats->astats.internal);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.resident,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
&astats->astats.resident);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.metadata_thp,
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
&astats->astats.metadata_thp);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
assert(atomic_load_zu(
&astats->astats.internal, ATOMIC_RELAXED) == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!destroyed) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
sdstats->allocated_small += astats->allocated_small;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
assert(astats->allocated_small == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
sdstats->nmalloc_small += astats->nmalloc_small;
sdstats->ndalloc_small += astats->ndalloc_small;
sdstats->nrequests_small += astats->nrequests_small;
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
sdstats->nfills_small += astats->nfills_small;
sdstats->nflushes_small += astats->nflushes_small;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.allocated_large,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
&astats->astats.allocated_large);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
assert(atomic_load_zu(&astats->astats.allocated_large,
ATOMIC_RELAXED) == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->astats.nmalloc_large,
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
&astats->astats.nmalloc_large);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->astats.ndalloc_large,
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
&astats->astats.ndalloc_large);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->astats.nrequests_large,
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
&astats->astats.nrequests_large);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->astats.nflushes_large,
Fix large.nflushes in the merged stats.
2019-08-27 04:18:50 +08:00
&astats->astats.nflushes_large);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(
&sdstats->astats.pa_shard_stats.abandoned_vm,
PA: Add a stats type.
2020-03-09 04:08:15 +08:00
&astats->astats.pa_shard_stats.abandoned_vm);
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->astats.tcache_bytes,
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
&astats->astats.tcache_bytes);
Added stats about number of bytes cached in tcache currently.
2016-04-23 09:37:44 +08:00
Add stats: arena uptime.
2017-05-17 04:56:00 +08:00
if (ctl_arena->arena_ind == 0) {
sdstats->astats.uptime = astats->astats.uptime;
}
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
/* Merge bin stats. */
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
for (i = 0; i < SC_NBINS; i++) {
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
bin_stats_t *bstats = &astats->bstats[i].stats_data;
bin_stats_t *merged = &sdstats->bstats[i].stats_data;
merged->nmalloc += bstats->nmalloc;
merged->ndalloc += bstats->ndalloc;
merged->nrequests += bstats->nrequests;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
merged->curregs += bstats->curregs;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
assert(bstats->curregs == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
merged->nfills += bstats->nfills;
merged->nflushes += bstats->nflushes;
merged->nslabs += bstats->nslabs;
merged->reslabs += bstats->reslabs;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
merged->curslabs += bstats->curslabs;
merged->nonfull_slabs += bstats->nonfull_slabs;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
assert(bstats->curslabs == 0);
assert(bstats->nonfull_slabs == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
malloc_mutex_prof_merge(&sdstats->bstats[i].mutex_data,
&astats->bstats[i].mutex_data);
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
/* Merge stats for large allocations. */
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
for (i = 0; i < SC_NSIZES - SC_NBINS; i++) {
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->lstats[i].nmalloc,
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
&astats->lstats[i].nmalloc);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->lstats[i].ndalloc,
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
&astats->lstats[i].ndalloc);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_locked_u64(&sdstats->lstats[i].nrequests,
Switch atomic uint64_ts in arena_stats_t to C11 atomics I expect this to be the trickiest conversion we will see, since we want atomics on 64-bit platforms, but are also always able to piggyback on some sort of external synchronization on non-64 bit platforms.
2017-03-09 07:56:31 +08:00
&astats->lstats[i].nrequests);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (!destroyed) {
sdstats->lstats[i].curlextents +=
astats->lstats[i].curlextents;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
assert(astats->lstats[i].curlextents == 0);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
}
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
/* Merge extents stats. */
for (i = 0; i < SC_NPSIZES; i++) {
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->estats[i].ndirty,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
&astats->estats[i].ndirty);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->estats[i].nmuzzy,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
&astats->estats[i].nmuzzy);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->estats[i].nretained,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
&astats->estats[i].nretained);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->estats[i].dirty_bytes,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
&astats->estats[i].dirty_bytes);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->estats[i].muzzy_bytes,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
&astats->estats[i].muzzy_bytes);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_accum_atomic_zu(&sdstats->estats[i].retained_bytes,
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
&astats->estats[i].retained_bytes);
}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
}
Push locks into arena bins. For bin-related allocation, protect data structures with bin locks rather than arena locks. Arena locks remain for run allocation/deallocation and other miscellaneous operations. Restructure statistics counters to maintain per bin allocated/nmalloc/ndalloc, but continue to provide arena-wide statistics via aggregation in the ctl code.
2010-03-14 12:32:56 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static void
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_refresh(tsdn_t *tsdn, arena_t *arena, ctl_arena_t *ctl_sdarena,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
unsigned i, bool destroyed) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_t *ctl_arena = arenas_i(i);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_clear(ctl_arena);
ctl_arena_stats_amerge(tsdn, ctl_arena, arena);
Fix stats.arenas.<i>.[...] for --disable-stats case. Add missing stats.arenas.<i>.{dss,lg_dirty_mult,decay_time} initialization. Fix stats.arenas.<i>.{pactive,pdirty} to read under the protection of the arena mutex.
2016-02-28 12:40:13 +08:00
/* Merge into sum stats as well. */
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_stats_sdmerge(ctl_sdarena, ctl_arena, destroyed);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
static unsigned
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
ctl_arena_init(tsd_t *tsd, extent_hooks_t *extent_hooks) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
unsigned arena_ind;
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_t *ctl_arena;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if ((ctl_arena = ql_last(&ctl_arenas->destroyed, destroyed_link)) !=
NULL) {
ql_remove(&ctl_arenas->destroyed, ctl_arena, destroyed_link);
arena_ind = ctl_arena->arena_ind;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arena_ind = ctl_arenas->narenas;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
/* Trigger stats allocation. */
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if (arenas_i_impl(tsd, arena_ind, false, true) == NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return UINT_MAX;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Convert all tsd variables to reside in a single tsd structure.
2014-09-23 12:09:23 +08:00
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
/* Initialize new arena. */
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if (arena_init(tsd_tsdn(tsd), arena_ind, extent_hooks) == NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return UINT_MAX;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (arena_ind == ctl_arenas->narenas) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arenas->narenas++;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return arena_ind;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
Add background thread related stats.
2017-05-13 03:30:33 +08:00
static void
ctl_background_thread_stats_read(tsdn_t *tsdn) {
background_thread_stats_t *stats = &ctl_stats->background_thread;
if (!have_background_thread ||
background_thread_stats_read(tsdn, stats)) {
memset(stats, 0, sizeof(background_thread_stats_t));
Initialization utilities for nstime
2019-12-17 04:41:06 +08:00
nstime_init_zero(&stats->run_interval);
Add background thread related stats.
2017-05-13 03:30:33 +08:00
}
Add max_per_bg_thd stats for per background thread mutexes. Added a new stats row to aggregate the maximum value of mutex counters for each background threads. Given that the per bg thd mutex is not expected to be contended, this counter is mainly for sanity check / debugging.
2019-08-15 07:10:09 +08:00
malloc_mutex_prof_copy(
&ctl_stats->mutex_prof_data[global_prof_mutex_max_per_bg_thd],
&stats->max_counter_per_bg_thd);
Add background thread related stats.
2017-05-13 03:30:33 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_refresh(tsdn_t *tsdn) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
unsigned i;
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_t *ctl_sarena = arenas_i(MALLCTL_ARENAS_ALL);
VARIABLE_ARRAY(arena_t *, tarenas, ctl_arenas->narenas);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/*
Fix some comments and whitespace.
2010-01-31 19:57:29 +08:00
* Clear sum stats, since they will be merged into by
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
* ctl_arena_refresh().
*/
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_clear(ctl_sarena);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
for (i = 0; i < ctl_arenas->narenas; i++) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
tarenas[i] = arena_get(tsdn, i, false);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
}
Refactor/fix arenas manipulation. Abstract arenas access to use arena_get() (or a0get() where appropriate) rather than directly reading e.g. arenas[ind]. Prior to the addition of the arenas.extend mallctl, the worst possible outcome of directly accessing arenas was a stale read, but arenas.extend may allocate and assign a new array to arenas. Add a tsd-based arenas_cache, which amortizes arenas reads. This introduces some subtle bootstrapping issues, with tsd_boot() now being split into tsd_boot[01]() to support tsd wrapper allocation bootstrapping, as well as an arenas_cache_bypass tsd variable which dynamically terminates allocation of arenas_cache itself. Promote a0malloc(), a0calloc(), and a0free() to be generally useful for internal allocation, and use them in several places (more may be appropriate). Abstract arena->nthreads management and fix a missing decrement during thread destruction (recent tsd refactoring left arenas_cleanup() unused). Change arena_choose() to propagate OOM, and handle OOM in all callers. This is important for providing consistent allocation behavior when the MALLOCX_ARENA() flag is being used. Prior to this fix, it was possible for an OOM to result in allocation silently allocating from a different arena than the one specified.
2014-10-08 14:14:57 +08:00
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
for (i = 0; i < ctl_arenas->narenas; i++) {
ctl_arena_t *ctl_arena = arenas_i(i);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
bool initialized = (tarenas[i] != NULL);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena->initialized = initialized;
if (initialized) {
ctl_arena_refresh(tsdn, tarenas[i], ctl_sarena, i,
false);
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
if (config_stats) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_stats->allocated = ctl_sarena->astats->allocated_small +
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
atomic_load_zu(&ctl_sarena->astats->astats.allocated_large,
ATOMIC_RELAXED);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_stats->active = (ctl_sarena->pactive << LG_PAGE);
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
ctl_stats->metadata = atomic_load_zu(
&ctl_sarena->astats->astats.base, ATOMIC_RELAXED) +
atomic_load_zu(&ctl_sarena->astats->astats.internal,
ATOMIC_RELAXED);
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
ctl_stats->metadata_thp = atomic_load_zu(
&ctl_sarena->astats->astats.metadata_thp, ATOMIC_RELAXED);
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
ctl_stats->resident = atomic_load_zu(
&ctl_sarena->astats->astats.resident, ATOMIC_RELAXED);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_stats->mapped = locked_read_atomic_zu(
PA: Move mapped into pa stats.
2020-03-10 03:06:19 +08:00
&ctl_sarena->astats->astats.pa_shard_stats.mapped);
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
ctl_stats->retained = locked_read_atomic_zu(
&ctl_sarena->astats->astats.retained);
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
Add background thread related stats.
2017-05-13 03:30:33 +08:00
ctl_background_thread_stats_read(tsdn);
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define READ_GLOBAL_MUTEX_PROF_DATA(i, mtx) \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
malloc_mutex_lock(tsdn, &mtx); \
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
malloc_mutex_prof_read(tsdn, &ctl_stats->mutex_prof_data[i], &mtx); \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
malloc_mutex_unlock(tsdn, &mtx);
Added JSON output for lock stats. Also added option 'x' to malloc_stats() to bypass lock section.
2017-03-12 17:28:52 +08:00
if (config_prof && opt_prof) {
Report stats for tdatas_mtx and prof_dump_mtx
2019-08-03 00:41:35 +08:00
READ_GLOBAL_MUTEX_PROF_DATA(
global_prof_mutex_prof, bt2gctx_mtx);
READ_GLOBAL_MUTEX_PROF_DATA(
global_prof_mutex_prof_thds_data, tdatas_mtx);
READ_GLOBAL_MUTEX_PROF_DATA(
global_prof_mutex_prof_dump, prof_dump_mtx);
Added JSON output for lock stats. Also added option 'x' to malloc_stats() to bypass lock section.
2017-03-12 17:28:52 +08:00
}
Add profiling for the background thread mutex.
2017-05-13 07:26:59 +08:00
if (have_background_thread) {
READ_GLOBAL_MUTEX_PROF_DATA(
global_prof_mutex_background_thread,
background_thread_lock);
} else {
memset(&ctl_stats->mutex_prof_data[
global_prof_mutex_background_thread], 0,
sizeof(mutex_prof_data_t));
}
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
/* We own ctl mutex already. */
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
malloc_mutex_prof_read(tsdn,
&ctl_stats->mutex_prof_data[global_prof_mutex_ctl],
&ctl_mtx);
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
#undef READ_GLOBAL_MUTEX_PROF_DATA
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arenas->epoch++;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
static bool
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
ctl_init(tsd_t *tsd) {
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
bool ret;
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
tsdn_t *tsdn = tsd_tsdn(tsd);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsdn, &ctl_mtx);
Convert to uniform style: cond == false --> !cond
2014-10-04 01:16:09 +08:00
if (!ctl_initialized) {
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_t *ctl_sarena, *ctl_darena;
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
unsigned i;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/*
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
* Allocate demand-zeroed space for pointers to the full
* range of supported arena indices.
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
*/
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if (ctl_arenas == NULL) {
ctl_arenas = (ctl_arenas_t *)base_alloc(tsdn,
b0get(), sizeof(ctl_arenas_t), QUANTUM);
if (ctl_arenas == NULL) {
ret = true;
goto label_return;
}
}
if (config_stats && ctl_stats == NULL) {
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
ctl_stats = (ctl_stats_t *)base_alloc(tsdn, b0get(),
sizeof(ctl_stats_t), QUANTUM);
if (ctl_stats == NULL) {
ret = true;
goto label_return;
}
}
/*
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
* Allocate space for the current full range of arenas
* here rather than doing it lazily elsewhere, in order
* to limit when OOM-caused errors can occur.
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
*/
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if ((ctl_sarena = arenas_i_impl(tsd, MALLCTL_ARENAS_ALL, false,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
true)) == NULL) {
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
ret = true;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_sarena->initialized = true;
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if ((ctl_darena = arenas_i_impl(tsd, MALLCTL_ARENAS_DESTROYED,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
false, true)) == NULL) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
ret = true;
goto label_return;
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_clear(ctl_darena);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
/*
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
* Don't toggle ctl_darena to initialized until an arena is
* actually destroyed, so that arena.<i>.initialized can be used
* to query whether the stats are relevant.
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
*/
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arenas->narenas = narenas_total_get();
for (i = 0; i < ctl_arenas->narenas; i++) {
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if (arenas_i_impl(tsd, i, false, true) == NULL) {
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
ret = true;
goto label_return;
}
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ql_new(&ctl_arenas->destroyed);
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
ctl_refresh(tsdn);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ctl_initialized = true;
}
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
ret = false;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsdn, &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
static int
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
ctl_lookup(tsdn_t *tsdn, const char *name, ctl_node_t const **nodesp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *mibp, size_t *depthp) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret;
const char *elm, *tdot, *dot;
size_t elen, i, j;
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
const ctl_named_node_t *node;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
elm = name;
/* Equivalent to strchrnul(). */
dot = ((tdot = strchr(elm, '.')) != NULL) ? tdot : strchr(elm, '\0');
elen = (size_t)((uintptr_t)dot - (uintptr_t)elm);
if (elen == 0) {
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
node = super_root_node;
for (i = 0; i < *depthp; i++) {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
assert(node);
assert(node->nchildren > 0);
if (ctl_named_node(node->children) != NULL) {
const ctl_named_node_t *pnode = node;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/* Children are named. */
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
for (j = 0; j < node->nchildren; j++) {
const ctl_named_node_t *child =
ctl_named_children(node, j);
if (strlen(child->name) == elen &&
strncmp(elm, child->name, elen) == 0) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
node = child;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (nodesp != NULL) {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
nodesp[i] =
(const ctl_node_t *)node;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
mibp[i] = j;
break;
}
}
if (node == pnode) {
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
} else {
Port to FreeBSD. Use FreeBSD-specific functions (_pthread_mutex_init_calloc_cb(), _malloc_{pre,post}fork()) to avoid bootstrapping issues due to allocation in libc and libthr. Add malloc_strtoumax() and use it instead of strtoul(). Disable validation code in malloc_vsnprintf() and malloc_strtoumax() until jemalloc is initialized. This is necessary because locale initialization causes allocation for both vsnprintf() and strtoumax(). Force the lazy-lock feature on in order to avoid pthread_self(), because it causes allocation. Use syscall(SYS_write, ...) rather than write(...), because libthr wraps write() and causes allocation. Without this workaround, it would not be possible to print error messages in malloc_conf_init() without substantially reworking bootstrapping. Fix choose_arena_hard() to look at how many threads are assigned to the candidate choice, rather than checking whether the arena is uninitialized. This bug potentially caused more arenas to be initialized than necessary.
2012-02-03 14:04:57 +08:00
uintmax_t index;
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
const ctl_indexed_node_t *inode;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/* Children are indexed. */
Port to FreeBSD. Use FreeBSD-specific functions (_pthread_mutex_init_calloc_cb(), _malloc_{pre,post}fork()) to avoid bootstrapping issues due to allocation in libc and libthr. Add malloc_strtoumax() and use it instead of strtoul(). Disable validation code in malloc_vsnprintf() and malloc_strtoumax() until jemalloc is initialized. This is necessary because locale initialization causes allocation for both vsnprintf() and strtoumax(). Force the lazy-lock feature on in order to avoid pthread_self(), because it causes allocation. Use syscall(SYS_write, ...) rather than write(...), because libthr wraps write() and causes allocation. Without this workaround, it would not be possible to print error messages in malloc_conf_init() without substantially reworking bootstrapping. Fix choose_arena_hard() to look at how many threads are assigned to the candidate choice, rather than checking whether the arena is uninitialized. This bug potentially caused more arenas to be initialized than necessary.
2012-02-03 14:04:57 +08:00
index = malloc_strtoumax(elm, NULL, 10);
if (index == UINTMAX_MAX || index > SIZE_T_MAX) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
inode = ctl_indexed_node(node->children);
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
node = inode->index(tsdn, mibp, *depthp, (size_t)index);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
if (node == NULL) {
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (nodesp != NULL) {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
nodesp[i] = (const ctl_node_t *)node;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
mibp[i] = (size_t)index;
}
if (node->ctl != NULL) {
/* Terminal node. */
if (*dot != '\0') {
/*
* The name contains more elements than are
* in this path through the tree.
*/
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
/* Complete lookup successful. */
*depthp = i + 1;
break;
}
/* Update elm. */
if (*dot == '\0') {
/* No more elements. */
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
elm = &dot[1];
dot = ((tdot = strchr(elm, '.')) != NULL) ? tdot :
strchr(elm, '\0');
elen = (size_t)((uintptr_t)dot - (uintptr_t)elm);
}
ret = 0;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
int
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
ctl_byname(tsd_t *tsd, const char *name, void *oldp, size_t *oldlenp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
void *newp, size_t newlen) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret;
size_t depth;
ctl_node_t const *nodes[CTL_MAX_DEPTH];
size_t mib[CTL_MAX_DEPTH];
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
const ctl_named_node_t *node;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if (!ctl_initialized && ctl_init(tsd)) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = EAGAIN;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
depth = CTL_MAX_DEPTH;
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
ret = ctl_lookup(tsd_tsdn(tsd), name, nodes, mib, &depth);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (ret != 0) {
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
node = ctl_named_node(nodes[depth-1]);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (node != NULL && node->ctl) {
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
ret = node->ctl(tsd, mib, depth, oldp, oldlenp, newp, newlen);
Small style fix in ctl.c
2017-03-02 06:43:35 +08:00
} else {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/* The name refers to a partial path through the ctl tree. */
ret = ENOENT;
}
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
return(ret);
}
int
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
ctl_nametomib(tsd_t *tsd, const char *name, size_t *mibp, size_t *miblenp) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret;
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if (!ctl_initialized && ctl_init(tsd)) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = EAGAIN;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
ret = ctl_lookup(tsd_tsdn(tsd), name, NULL, mibp, miblenp);
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
return(ret);
}
int
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
ctl_bymib(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret;
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
const ctl_named_node_t *node;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
size_t i;
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if (!ctl_initialized && ctl_init(tsd)) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = EAGAIN;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
/* Iterate down the tree. */
node = super_root_node;
for (i = 0; i < miblen; i++) {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
assert(node);
assert(node->nchildren > 0);
if (ctl_named_node(node->children) != NULL) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/* Children are named. */
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
if (node->nchildren <= mib[i]) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
node = ctl_named_children(node, mib[i]);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
} else {
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
const ctl_indexed_node_t *inode;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/* Indexed element. */
Avoid using a union for ctl_node_s MSVC doesn't support C99, and as such doesn't support designated initialization of structs and unions. As there is never a mix of indexed and named nodes, it is pretty straightforward to use a different type for each.
2012-04-20 14:38:42 +08:00
inode = ctl_indexed_node(node->children);
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
node = inode->index(tsd_tsdn(tsd), mib, miblen, mib[i]);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
if (node == NULL) {
ret = ENOENT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
}
}
/* Call the ctl function. */
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (node && node->ctl) {
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
ret = node->ctl(tsd, mib, miblen, oldp, oldlenp, newp, newlen);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/* Partial MIB. */
ret = ENOENT;
}
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
return(ret);
}
bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_boot(void) {
Allow mutexes to take a lock ordering enum at construction. This lets us specify whether and how mutexes of the same rank are allowed to be acquired. Currently, we only allow two polices (only a single mutex at a given rank at a time, and mutexes acquired in ascending order), but we can plausibly allow more (e.g. the "release uncontended mutexes before blocking").
2017-05-16 06:38:15 +08:00
if (malloc_mutex_init(&ctl_mtx, "ctl", WITNESS_RANK_CTL,
malloc_mutex_rank_exclusive)) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return true;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ctl_initialized = false;
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_prefork(tsdn_t *tsdn) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_prefork(tsdn, &ctl_mtx);
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
}
void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_postfork_parent(tsdn_t *tsdn) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_postfork_parent(tsdn, &ctl_mtx);
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
}
void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
ctl_postfork_child(tsdn_t *tsdn) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_postfork_child(tsdn, &ctl_mtx);
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
}
Add "prof.dump_prefix" to override filename prefixes for dumps.
2019-09-10 11:18:41 +08:00
void
ctl_mtx_assert_held(tsdn_t *tsdn) {
malloc_mutex_assert_owner(tsdn, &ctl_mtx);
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/******************************************************************************/
/* *_ctl() functions. */
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define READONLY() do { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
if (newp != NULL || newlen != 0) { \
ret = EPERM; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
goto label_return; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
} \
} while (0)
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define WRITEONLY() do { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
if (oldp != NULL || oldlenp != NULL) { \
ret = EPERM; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
goto label_return; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
} \
} while (0)
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
/* Can read or write, but not both. */
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define READ_XOR_WRITE() do { \
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
if ((oldp != NULL && oldlenp != NULL) && (newp != NULL || \
newlen != 0)) { \
ret = EPERM; \
goto label_return; \
} \
} while (0)
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
/* Can neither read nor write. */
#define NEITHER_READ_NOR_WRITE() do { \
if (oldp != NULL || oldlenp != NULL || newp != NULL || \
newlen != 0) { \
ret = EPERM; \
goto label_return; \
} \
} while (0)
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define READ(v, t) do { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
if (oldp != NULL && oldlenp != NULL) { \
if (*oldlenp != sizeof(t)) { \
size_t copylen = (sizeof(t) <= *oldlenp) \
? sizeof(t) : *oldlenp; \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
memcpy(oldp, (void *)&(v), copylen); \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = EINVAL; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
goto label_return; \
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
} \
*(t *)oldp = (v); \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
} \
} while (0)
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define WRITE(v, t) do { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
if (newp != NULL) { \
if (newlen != sizeof(t)) { \
ret = EINVAL; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
goto label_return; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
} \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
(v) = *(t *)newp; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
} \
} while (0)
Add ASSURED_WRITE in mallctl
2020-02-05 08:36:02 +08:00
#define ASSURED_WRITE(v, t) do { \
if (newp == NULL || newlen != sizeof(t)) { \
ret = EINVAL; \
goto label_return; \
} \
(v) = *(t *)newp; \
} while (0)
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define MIB_UNSIGNED(v, i) do { \
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
if (mib[i] > UINT_MAX) { \
ret = EFAULT; \
goto label_return; \
} \
v = (unsigned)mib[i]; \
} while (0)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
/*
* There's a lot of code duplication in the following macros due to limitations
* in how nested cpp macros are expanded.
*/
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_RO_CLGEN(c, l, n, v, t) \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
static int \
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp, \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) { \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
int ret; \
t oldval; \
\
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!(c)) { \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT; \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} \
if (l) { \
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx); \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
READONLY(); \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
oldval = (v); \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
READ(oldval, t); \
\
ret = 0; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
label_return: \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (l) { \
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx); \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret; \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_RO_CGEN(c, n, v, t) \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
static int \
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, \
void *oldp, size_t *oldlenp, void *newp, size_t newlen) { \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
int ret; \
t oldval; \
\
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!(c)) { \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT; \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} \
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx); \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
READONLY(); \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
oldval = (v); \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
READ(oldval, t); \
\
ret = 0; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
label_return: \
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx); \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret; \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_RO_GEN(n, v, t) \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static int \
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp, \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) { \
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
int ret; \
t oldval; \
\
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx); \
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
READONLY(); \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
oldval = (v); \
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
READ(oldval, t); \
\
ret = 0; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
label_return: \
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx); \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret; \
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
}
/*
* ctl_mtx is not acquired, under the assumption that no pertinent data will
* mutate during the call.
*/
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_RO_NL_CGEN(c, n, v, t) \
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
static int \
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, \
void *oldp, size_t *oldlenp, void *newp, size_t newlen) { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret; \
t oldval; \
\
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!(c)) { \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT; \
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
READONLY(); \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
oldval = (v); \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
READ(oldval, t); \
\
ret = 0; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
label_return: \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_RO_NL_GEN(n, v, t) \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static int \
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, \
void *oldp, size_t *oldlenp, void *newp, size_t newlen) { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret; \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
t oldval; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
\
READONLY(); \
Fix "arenas.extend" mallctl to return the number of arenas. Reported by Mike Hommey.
2012-11-30 14:13:04 +08:00
oldval = (v); \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
READ(oldval, t); \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
\
ret = 0; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
label_return: \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define CTL_RO_CONFIG_GEN(n, t) \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static int \
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
n##_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, \
void *oldp, size_t *oldlenp, void *newp, size_t newlen) { \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret; \
Add --with-malloc-conf. Add --with-malloc-conf, which makes it possible to embed a default options string during configuration.
2016-02-08 06:23:22 +08:00
t oldval; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
\
READONLY(); \
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
oldval = n; \
Add --with-malloc-conf. Add --with-malloc-conf, which makes it possible to embed a default options string during configuration.
2016-02-08 06:23:22 +08:00
READ(oldval, t); \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
\
ret = 0; \
Fix the "epoch" mallctl. Fix the "epoch" mallctl to update cached stats even if the passed in epoch is 0.
2012-04-26 04:12:46 +08:00
label_return: \
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret; \
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
/******************************************************************************/
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
CTL_RO_NL_GEN(version, JEMALLOC_VERSION, const char *)
Add release versioning support. Base version string on 'git describe --long', and provide cpp macros in jemalloc.h. Add the version mallctl.
2010-03-03 05:01:16 +08:00
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
epoch_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
int ret;
Silence an unused variable warning. Reported by Ricardo Nabinger Sanchez.
2013-10-20 08:19:49 +08:00
UNUSED uint64_t newval;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
WRITE(newval, uint64_t);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (newp != NULL) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
ctl_refresh(tsd_tsdn(tsd));
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
READ(ctl_arenas->epoch, uint64_t);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
ret = 0;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
background_thread_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp,
void *newp, size_t newlen) {
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
int ret;
bool oldval;
if (!have_background_thread) {
return ENOENT;
}
Set isthreaded when enabling background_thread.
2017-06-02 03:52:09 +08:00
background_thread_ctl_init(tsd_tsdn(tsd));
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
Drop high rank locks when creating threads. Avoid holding arenas_lock and background_thread_lock when creating background threads, because pthread_create may take internal locks, and potentially cause deadlock with jemalloc internal locks.
2017-06-08 06:49:09 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &background_thread_lock);
if (newp == NULL) {
oldval = background_thread_enabled();
READ(oldval, bool);
} else {
if (newlen != sizeof(bool)) {
ret = EINVAL;
goto label_return;
}
oldval = background_thread_enabled();
READ(oldval, bool);
bool newval = *(bool *)newp;
if (newval == oldval) {
ret = 0;
goto label_return;
}
background_thread_enabled_set(tsd_tsdn(tsd), newval);
if (newval) {
if (background_threads_enable(tsd)) {
ret = EFAULT;
goto label_return;
}
} else {
if (background_threads_disable(tsd)) {
ret = EFAULT;
goto label_return;
}
}
}
ret = 0;
label_return:
malloc_mutex_unlock(tsd_tsdn(tsd), &background_thread_lock);
Drop high rank locks when creating threads. Avoid holding arenas_lock and background_thread_lock when creating background threads, because pthread_create may take internal locks, and potentially cause deadlock with jemalloc internal locks.
2017-06-08 06:49:09 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
return ret;
}
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
max_background_threads_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
int ret;
size_t oldval;
if (!have_background_thread) {
return ENOENT;
}
background_thread_ctl_init(tsd_tsdn(tsd));
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
malloc_mutex_lock(tsd_tsdn(tsd), &background_thread_lock);
if (newp == NULL) {
oldval = max_background_threads;
READ(oldval, size_t);
} else {
if (newlen != sizeof(size_t)) {
ret = EINVAL;
goto label_return;
}
oldval = max_background_threads;
READ(oldval, size_t);
size_t newval = *(size_t *)newp;
if (newval == oldval) {
ret = 0;
goto label_return;
}
if (newval > opt_max_background_threads) {
ret = EINVAL;
goto label_return;
}
if (background_thread_enabled()) {
background_thread_enabled_set(tsd_tsdn(tsd), false);
if (background_threads_disable(tsd)) {
ret = EFAULT;
goto label_return;
}
max_background_threads = newval;
background_thread_enabled_set(tsd_tsdn(tsd), true);
if (background_threads_enable(tsd)) {
ret = EFAULT;
goto label_return;
}
} else {
max_background_threads = newval;
}
}
ret = 0;
label_return:
malloc_mutex_unlock(tsd_tsdn(tsd), &background_thread_lock);
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
return ret;
}
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
/******************************************************************************/
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Add --with-malloc-conf. Add --with-malloc-conf, which makes it possible to embed a default options string during configuration.
2016-02-08 06:23:22 +08:00
CTL_RO_CONFIG_GEN(config_cache_oblivious, bool)
CTL_RO_CONFIG_GEN(config_debug, bool)
CTL_RO_CONFIG_GEN(config_fill, bool)
CTL_RO_CONFIG_GEN(config_lazy_lock, bool)
CTL_RO_CONFIG_GEN(config_malloc_conf, const char *)
Safety checks: Expose config value via mallctl and stats.
2019-03-23 08:13:45 +08:00
CTL_RO_CONFIG_GEN(config_opt_safety_checks, bool)
Add --with-malloc-conf. Add --with-malloc-conf, which makes it possible to embed a default options string during configuration.
2016-02-08 06:23:22 +08:00
CTL_RO_CONFIG_GEN(config_prof, bool)
CTL_RO_CONFIG_GEN(config_prof_libgcc, bool)
CTL_RO_CONFIG_GEN(config_prof_libunwind, bool)
CTL_RO_CONFIG_GEN(config_stats, bool)
CTL_RO_CONFIG_GEN(config_utrace, bool)
CTL_RO_CONFIG_GEN(config_xmalloc, bool)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
/******************************************************************************/
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_NL_GEN(opt_abort, opt_abort, bool)
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
CTL_RO_NL_GEN(opt_abort_conf, opt_abort_conf, bool)
Add confirm_conf option If the confirm_conf option is set, when the program starts, each of the four malloc_conf strings will be printed, and each option will be printed when being set.
2019-05-01 04:54:00 +08:00
CTL_RO_NL_GEN(opt_confirm_conf, opt_confirm_conf, bool)
Change opt.metadata_thp to [disabled,auto,always]. To avoid the high RSS caused by THP + low usage arena (i.e. THP becomes a significant percentage), added a new "auto" option which will only start using THP after a base allocator used up the first THP region. Starting from the second hugepage (in a single arena), "auto" behaves the same as "always", i.e. madvise hugepage right away.
2017-08-25 05:29:28 +08:00
CTL_RO_NL_GEN(opt_metadata_thp, metadata_thp_mode_names[opt_metadata_thp],
const char *)
Refactor !opt.munmap to opt.retain.
2017-04-27 07:26:12 +08:00
CTL_RO_NL_GEN(opt_retain, opt_retain, bool)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_NL_GEN(opt_dss, opt_dss, const char *)
Make opt_narenas unsigned rather than size_t.
2016-02-25 03:03:40 +08:00
CTL_RO_NL_GEN(opt_narenas, opt_narenas, unsigned)
Refactor/fix background_thread/percpu_arena bootstrapping. Refactor bootstrapping such that dlsym() is called during the bootstrapping phase that can tolerate reentrant allocation.
2017-06-01 07:45:14 +08:00
CTL_RO_NL_GEN(opt_percpu_arena, percpu_arena_mode_names[opt_percpu_arena],
const char *)
Rename huge_threshold to oversize_threshold. The keyword huge tend to remind people of huge pages which is not relevent to the feature.
2019-01-25 08:15:04 +08:00
CTL_RO_NL_GEN(opt_oversize_threshold, opt_oversize_threshold, size_t)
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
CTL_RO_NL_GEN(opt_background_thread, opt_background_thread, bool)
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
CTL_RO_NL_GEN(opt_max_background_threads, opt_max_background_threads, size_t)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_RO_NL_GEN(opt_dirty_decay_ms, opt_dirty_decay_ms, ssize_t)
CTL_RO_NL_GEN(opt_muzzy_decay_ms, opt_muzzy_decay_ms, ssize_t)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_NL_GEN(opt_stats_print, opt_stats_print, bool)
Add opt.stats_print_opts. The value is passed to atexit(3)-triggered malloc_stats_print() calls.
2017-05-28 06:35:36 +08:00
CTL_RO_NL_GEN(opt_stats_print_opts, opt_stats_print_opts, const char *)
Implement opt.stats_interval and the _opts options. Add options stats_interval and stats_interval_opts to allow interval based stats printing. This provides an easy way to collect stats without code changes, because opt.stats_print may not work (some binaries never exit).
2020-01-14 14:29:17 +08:00
CTL_RO_NL_GEN(opt_stats_interval, opt_stats_interval, int64_t)
CTL_RO_NL_GEN(opt_stats_interval_opts, opt_stats_interval_opts, const char *)
Introduce two new modes of junk filling: "alloc" and "free". In addition to true/false, opt.junk can now be either "alloc" or "free", giving applications the possibility of junking memory only on allocation or deallocation. This resolves #172.
2014-12-09 05:12:41 +08:00
CTL_RO_NL_CGEN(config_fill, opt_junk, opt_junk, const char *)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_NL_CGEN(config_fill, opt_zero, opt_zero, bool)
CTL_RO_NL_CGEN(config_utrace, opt_utrace, opt_utrace, bool)
CTL_RO_NL_CGEN(config_xmalloc, opt_xmalloc, opt_xmalloc, bool)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_NL_GEN(opt_tcache, opt_tcache, bool)
Add opt.thp which allows explicit hugepage usage. "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
2018-02-17 06:19:19 +08:00
CTL_RO_NL_GEN(opt_thp, thp_mode_names[opt_thp], const char *)
Add opt.lg_extent_max_active_fit When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
2017-11-10 05:51:39 +08:00
CTL_RO_NL_GEN(opt_lg_extent_max_active_fit, opt_lg_extent_max_active_fit,
size_t)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_NL_GEN(opt_lg_tcache_max, opt_lg_tcache_max, ssize_t)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_NL_CGEN(config_prof, opt_prof, opt_prof, bool)
CTL_RO_NL_CGEN(config_prof, opt_prof_prefix, opt_prof_prefix, const char *)
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
CTL_RO_NL_CGEN(config_prof, opt_prof_active, opt_prof_active, bool)
CTL_RO_NL_CGEN(config_prof, opt_prof_thread_active_init,
opt_prof_thread_active_init, bool)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_NL_CGEN(config_prof, opt_lg_prof_sample, opt_lg_prof_sample, size_t)
CTL_RO_NL_CGEN(config_prof, opt_prof_accum, opt_prof_accum, bool)
CTL_RO_NL_CGEN(config_prof, opt_lg_prof_interval, opt_lg_prof_interval, ssize_t)
CTL_RO_NL_CGEN(config_prof, opt_prof_gdump, opt_prof_gdump, bool)
CTL_RO_NL_CGEN(config_prof, opt_prof_final, opt_prof_final, bool)
CTL_RO_NL_CGEN(config_prof, opt_prof_leak, opt_prof_leak, bool)
Last-N profiling mode
2019-12-19 05:38:14 +08:00
CTL_RO_NL_CGEN(config_prof, opt_prof_recent_alloc_max,
opt_prof_recent_alloc_max, ssize_t)
Add option to fetch system thread name on each prof sample
2020-03-25 08:53:41 +08:00
CTL_RO_NL_CGEN(config_prof, opt_prof_experimental_use_sys_thread_name,
opt_prof_experimental_use_sys_thread_name, bool)
Realloc: Make behavior of realloc(ptr, 0) configurable.
2019-09-24 08:56:19 +08:00
CTL_RO_NL_GEN(opt_zero_realloc,
zero_realloc_mode_names[opt_zero_realloc_action], const char *)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
/******************************************************************************/
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
thread_arena_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
int ret;
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
arena_t *oldarena;
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
unsigned newind, oldind;
Fix huge_palloc() regression. Split arena_choose() into arena_[i]choose() and use arena_ichoose() for arena lookup during internal allocation. This fixes huge_palloc() so that it always succeeds during extent node allocation. This regression was introduced by 66cd953514a18477eb49732e40d5c2ab5f1b12c5 (Do not allocate metadata via non-auto arenas, nor tcaches.).
2016-05-04 06:00:42 +08:00
oldarena = arena_choose(tsd, NULL);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (oldarena == NULL) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return EAGAIN;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
newind = oldind = arena_ind_get(oldarena);
Fix a "thread.arena" mallctl bug. Fix a variable reversal bug in mallctl("thread.arena", ...).
2011-03-15 02:39:49 +08:00
WRITE(newind, unsigned);
READ(oldind, unsigned);
Implement per-CPU arena. The new feature, opt.percpu_arena, determines thread-arena association dynamically based CPU id. Three modes are supported: "percpu", "phycpu" and disabled. "percpu" uses the current core id (with help from sched_getcpu()) directly as the arena index, while "phycpu" will assign threads on the same physical CPU to the same arena. In other words, "percpu" means # of arenas == # of CPUs, while "phycpu" has # of arenas == 1/2 * (# of CPUs). Note that no runtime check on whether hyper threading is enabled is added yet. When enabled, threads will be migrated between arenas when a CPU change is detected. In the current design, to reduce overhead from reading CPU id, each arena tracks the thread accessed most recently. When a new thread comes in, we will read CPU id and update arena if necessary.
2017-02-03 09:02:05 +08:00
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
if (newind != oldind) {
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
arena_t *newarena;
Reduce "thread.arena" mallctl contention. This resolves #460.
2016-10-04 01:37:12 +08:00
if (newind >= narenas_total_get()) {
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
/* New arena index is out of range. */
ret = EFAULT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
}
Implement per-CPU arena. The new feature, opt.percpu_arena, determines thread-arena association dynamically based CPU id. Three modes are supported: "percpu", "phycpu" and disabled. "percpu" uses the current core id (with help from sched_getcpu()) directly as the arena index, while "phycpu" will assign threads on the same physical CPU to the same arena. In other words, "percpu" means # of arenas == # of CPUs, while "phycpu" has # of arenas == 1/2 * (# of CPUs). Note that no runtime check on whether hyper threading is enabled is added yet. When enabled, threads will be migrated between arenas when a CPU change is detected. In the current design, to reduce overhead from reading CPU id, each arena tracks the thread accessed most recently. When a new thread comes in, we will read CPU id and update arena if necessary.
2017-02-03 09:02:05 +08:00
if (have_percpu_arena &&
Refactor/fix background_thread/percpu_arena bootstrapping. Refactor bootstrapping such that dlsym() is called during the bootstrapping phase that can tolerate reentrant allocation.
2017-06-01 07:45:14 +08:00
PERCPU_ARENA_ENABLED(opt_percpu_arena)) {
if (newind < percpu_arena_ind_limit(opt_percpu_arena)) {
Implement per-CPU arena. The new feature, opt.percpu_arena, determines thread-arena association dynamically based CPU id. Three modes are supported: "percpu", "phycpu" and disabled. "percpu" uses the current core id (with help from sched_getcpu()) directly as the arena index, while "phycpu" will assign threads on the same physical CPU to the same arena. In other words, "percpu" means # of arenas == # of CPUs, while "phycpu" has # of arenas == 1/2 * (# of CPUs). Note that no runtime check on whether hyper threading is enabled is added yet. When enabled, threads will be migrated between arenas when a CPU change is detected. In the current design, to reduce overhead from reading CPU id, each arena tracks the thread accessed most recently. When a new thread comes in, we will read CPU id and update arena if necessary.
2017-02-03 09:02:05 +08:00
/*
* If perCPU arena is enabled, thread_arena
* control is not allowed for the auto arena
* range.
*/
ret = EPERM;
goto label_return;
}
}
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
/* Initialize arena if necessary. */
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
newarena = arena_get(tsd_tsdn(tsd), newind, true);
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
if (newarena == NULL) {
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
ret = EAGAIN;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
}
Refactor/fix arenas manipulation. Abstract arenas access to use arena_get() (or a0get() where appropriate) rather than directly reading e.g. arenas[ind]. Prior to the addition of the arenas.extend mallctl, the worst possible outcome of directly accessing arenas was a stale read, but arenas.extend may allocate and assign a new array to arenas. Add a tsd-based arenas_cache, which amortizes arenas reads. This introduces some subtle bootstrapping issues, with tsd_boot() now being split into tsd_boot[01]() to support tsd wrapper allocation bootstrapping, as well as an arenas_cache_bypass tsd variable which dynamically terminates allocation of arenas_cache itself. Promote a0malloc(), a0calloc(), and a0free() to be generally useful for internal allocation, and use them in several places (more may be appropriate). Abstract arena->nthreads management and fix a missing decrement during thread destruction (recent tsd refactoring left arenas_cleanup() unused). Change arena_choose() to propagate OOM, and handle OOM in all callers. This is important for providing consistent allocation behavior when the MALLOCX_ARENA() flag is being used. Prior to this fix, it was possible for an OOM to result in allocation silently allocating from a different arena than the one specified.
2014-10-08 14:14:57 +08:00
/* Set new arena/tcache associations. */
arena_migrate(tsd, oldind, newind);
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
if (tcache_available(tsd)) {
tcache_arena_reassociate(tsd_tsdn(tsd),
tsd_tcachep_get(tsd), newarena);
Fix a "thread.arena" mallctl bug. When setting a new arena association for the calling thread, also update the tcache's cached arena pointer, primarily so that tcache_alloc_small_hard() uses the intended arena.
2010-12-30 04:21:05 +08:00
}
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
}
ret = 0;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add the thread.arena mallctl. Make it possible for each thread to manage which arena it is associated with. Implement the 'tests' and 'check' build targets.
2010-08-14 06:42:29 +08:00
}
Pull thread_(de)allocated out of config_stats
2019-08-13 02:03:36 +08:00
CTL_RO_NL_GEN(thread_allocated, tsd_thread_allocated_get(tsd), uint64_t)
CTL_RO_NL_GEN(thread_allocatedp, tsd_thread_allocatedp_get(tsd), uint64_t *)
CTL_RO_NL_GEN(thread_deallocated, tsd_thread_deallocated_get(tsd), uint64_t)
CTL_RO_NL_GEN(thread_deallocatedp, tsd_thread_deallocatedp_get(tsd), uint64_t *)
Add per thread allocation counters, and enhance heap sampling. Add the "thread.allocated" and "thread.deallocated" mallctls, which can be used to query the total number of bytes ever allocated/deallocated by the calling thread. Add s2u() and sa2u(), which can be used to compute the usable size that will result from an allocation request of a particular size/alignment. Re-factor ipalloc() to use sa2u(). Enhance the heap profiler to trigger samples based on usable size, rather than request size. This has a subtle, but important, impact on the accuracy of heap sampling. For example, previous to this change, 16- and 17-byte objects were sampled at nearly the same rate, but 17-byte objects actually consume 32 bytes each. Therefore it was possible for the sample to be somewhat skewed compared to actual memory usage of the allocated objects.
2010-10-21 08:39:18 +08:00
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
thread_tcache_enabled_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
int ret;
bool oldval;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Integrate auto tcache into TSD. The embedded tcache is initialized upon tsd initialization. The avail arrays for the tbins will be allocated / deallocated accordingly during init / cleanup. With this change, the pointer to the auto tcache will always be available, as long as we have access to the TSD. tcache_available() (called in tcache_get()) is provided to check if we should use tcache.
2017-03-28 12:50:38 +08:00
oldval = tcache_enabled_get(tsd);
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
if (newp != NULL) {
if (newlen != sizeof(bool)) {
ret = EINVAL;
goto label_return;
}
Integrate auto tcache into TSD. The embedded tcache is initialized upon tsd initialization. The avail arrays for the tbins will be allocated / deallocated accordingly during init / cleanup. With this change, the pointer to the auto tcache will always be available, as long as we have access to the TSD. tcache_available() (called in tcache_get()) is provided to check if we should use tcache.
2017-03-28 12:50:38 +08:00
tcache_enabled_set(tsd, *(bool *)newp);
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
}
READ(oldval, bool);
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
}
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
thread_tcache_flush_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
int ret;
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
if (!tcache_available(tsd)) {
ret = EFAULT;
goto label_return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
NEITHER_READ_NOR_WRITE();
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
Pass tsd to tcache_flush().
2017-06-16 06:16:18 +08:00
tcache_flush(tsd);
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
thread_prof_name_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
int ret;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
READ_XOR_WRITE();
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
if (newp != NULL) {
if (newlen != sizeof(const char *)) {
ret = EINVAL;
goto label_return;
}
Convert all tsd variables to reside in a single tsd structure.
2014-09-23 12:09:23 +08:00
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
if ((ret = prof_thread_name_set(tsd, *(const char **)newp)) !=
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
0) {
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
goto label_return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
} else {
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
const char *oldname = prof_thread_name_get(tsd);
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
READ(oldname, const char *);
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
}
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
}
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
thread_prof_active_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
int ret;
bool oldval;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
oldval = prof_thread_active_get(tsd);
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
if (newp != NULL) {
if (newlen != sizeof(bool)) {
ret = EINVAL;
goto label_return;
}
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
if (prof_thread_active_set(tsd, *(bool *)newp)) {
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
ret = EAGAIN;
goto label_return;
}
}
READ(oldval, bool);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
}
Add "thread.idle" mallctl. This can encapsulate various internal cleaning logic, and can be used to free up resources before a long sleep.
2020-01-23 03:13:26 +08:00
static int
thread_idle_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
int ret;
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
NEITHER_READ_NOR_WRITE();
Add "thread.idle" mallctl. This can encapsulate various internal cleaning logic, and can be used to free up resources before a long sleep.
2020-01-23 03:13:26 +08:00
if (tcache_available(tsd)) {
tcache_flush(tsd);
}
/*
* This heuristic is perhaps not the most well-considered. But it
* matches the only idling policy we have experience with in the status
* quo. Over time we should investigate more principled approaches.
*/
if (opt_narenas > ncpus * 2) {
arena_t *arena = arena_choose(tsd, NULL);
if (arena != NULL) {
arena_decay(tsd_tsdn(tsd), arena, false, true);
}
/*
* The missing arena case is not actually an error; a thread
* might be idle before it associates itself to one. This is
* unusual, but not wrong.
*/
}
ret = 0;
label_return:
return ret;
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
/******************************************************************************/
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
tcache_create_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
int ret;
unsigned tcache_ind;
READONLY();
Break prof and tcache knowledge of b0.
2020-02-18 06:09:29 +08:00
if (tcaches_create(tsd, b0get(), &tcache_ind)) {
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
ret = EFAULT;
goto label_return;
}
READ(tcache_ind, unsigned);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
}
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
tcache_flush_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
int ret;
unsigned tcache_ind;
WRITEONLY();
Add ASSURED_WRITE in mallctl
2020-02-05 08:36:02 +08:00
ASSURED_WRITE(tcache_ind, unsigned);
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
tcaches_flush(tsd, tcache_ind);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
}
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
tcache_destroy_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
int ret;
unsigned tcache_ind;
WRITEONLY();
Add ASSURED_WRITE in mallctl
2020-02-05 08:36:02 +08:00
ASSURED_WRITE(tcache_ind, unsigned);
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
tcaches_destroy(tsd, tcache_ind);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement explicit tcache support. Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
2015-01-30 07:30:47 +08:00
}
/******************************************************************************/
Replace the arenas.initialized mallctl with arena.<i>.initialized .
2017-01-05 02:21:53 +08:00
static int
arena_i_initialized_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Replace the arenas.initialized mallctl with arena.<i>.initialized .
2017-01-05 02:21:53 +08:00
int ret;
tsdn_t *tsdn = tsd_tsdn(tsd);
unsigned arena_ind;
bool initialized;
READONLY();
MIB_UNSIGNED(arena_ind, 1);
malloc_mutex_lock(tsdn, &ctl_mtx);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
initialized = arenas_i(arena_ind)->initialized;
Replace the arenas.initialized mallctl with arena.<i>.initialized .
2017-01-05 02:21:53 +08:00
malloc_mutex_unlock(tsdn, &ctl_mtx);
READ(initialized, bool);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Replace the arenas.initialized mallctl with arena.<i>.initialized .
2017-01-05 02:21:53 +08:00
}
Fix deadlock in the arenas.purge mallctl. Fix deadlock in the arenas.purge mallctl due to recursive mutex acquisition.
2012-11-04 12:18:28 +08:00
static void
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
arena_i_decay(tsdn_t *tsdn, unsigned arena_ind, bool all) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsdn, &ctl_mtx);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
{
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
unsigned narenas = ctl_arenas->narenas;
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
/*
* Access via index narenas is deprecated, and scheduled for
* removal in 6.0.0.
*/
if (arena_ind == MALLCTL_ARENAS_ALL || arena_ind == narenas) {
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
unsigned i;
VARIABLE_ARRAY(arena_t *, tarenas, narenas);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
for (i = 0; i < narenas; i++) {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
tarenas[i] = arena_get(tsdn, i, false);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
/*
* No further need to hold ctl_mtx, since narenas and
* tarenas contain everything needed below.
*/
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsdn, &ctl_mtx);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
for (i = 0; i < narenas; i++) {
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (tarenas[i] != NULL) {
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_decay(tsdn, tarenas[i], false,
all);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
}
} else {
arena_t *tarena;
assert(arena_ind < narenas);
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
tarena = arena_get(tsdn, arena_ind, false);
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
/* No further need to hold ctl_mtx. */
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsdn, &ctl_mtx);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (tarena != NULL) {
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_decay(tsdn, tarena, false, all);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
}
}
static int
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
arena_i_decay_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) {
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
int ret;
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
unsigned arena_ind;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
NEITHER_READ_NOR_WRITE();
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
MIB_UNSIGNED(arena_ind, 1);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
arena_i_decay(tsd_tsdn(tsd), arena_ind, false);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
}
static int
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
arena_i_purge_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) {
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
int ret;
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
unsigned arena_ind;
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
NEITHER_READ_NOR_WRITE();
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
MIB_UNSIGNED(arena_ind, 1);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
arena_i_decay(tsd_tsdn(tsd), arena_ind, true);
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Fix deadlock in the arenas.purge mallctl. Fix deadlock in the arenas.purge mallctl due to recursive mutex acquisition.
2012-11-04 12:18:28 +08:00
ret = 0;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
static int
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
arena_i_reset_destroy_helper(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen, unsigned *arena_ind,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arena_t **arena) {
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
int ret;
Introduce NEITHER_READ_NOR_WRITE in ctl. This is slightly clearer in meaning. A function that is both READONLY() and WRITEONLY() is in fact neither one.
2020-01-23 06:59:28 +08:00
NEITHER_READ_NOR_WRITE();
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
MIB_UNSIGNED(*arena_ind, 1);
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
*arena = arena_get(tsd_tsdn(tsd), *arena_ind, false);
Bypass extent tracking for auto arenas. Tracking extents is required by arena_reset. To support this, the extent linkage was used for tracking 1) large allocations, and 2) full slabs. However modifying the extent linkage could be an expensive operation as it likely incurs cache misses. Since we forbid arena_reset on auto arenas, let's bypass the linkage operations for auto arenas.
2017-04-21 06:19:02 +08:00
if (*arena == NULL || arena_is_auto(*arena)) {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
ret = EFAULT;
goto label_return;
}
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
}
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
static void
arena_reset_prepare_background_thread(tsd_t *tsd, unsigned arena_ind) {
/* Temporarily disable the background thread during arena reset. */
if (have_background_thread) {
malloc_mutex_lock(tsd_tsdn(tsd), &background_thread_lock);
if (background_thread_enabled()) {
background_thread_info_t *info =
Fix background thread index issues with max_background_threads.
2018-05-12 07:32:29 +08:00
background_thread_info_get(arena_ind);
Combine background_thread started / paused into state.
2017-06-10 06:45:25 +08:00
assert(info->state == background_thread_started);
Move background thread creation to background_thread_0. To avoid complications, avoid invoking pthread_create "internally", instead rely on thread0 to launch new threads, and also terminating threads when asked.
2017-06-09 13:46:31 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &info->mtx);
Combine background_thread started / paused into state.
2017-06-10 06:45:25 +08:00
info->state = background_thread_paused;
Move background thread creation to background_thread_0. To avoid complications, avoid invoking pthread_create "internally", instead rely on thread0 to launch new threads, and also terminating threads when asked.
2017-06-09 13:46:31 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &info->mtx);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
}
}
}
static void
arena_reset_finish_background_thread(tsd_t *tsd, unsigned arena_ind) {
if (have_background_thread) {
if (background_thread_enabled()) {
background_thread_info_t *info =
Fix background thread index issues with max_background_threads.
2018-05-12 07:32:29 +08:00
background_thread_info_get(arena_ind);
Fix assertion typos. Reported by Conrad Meyer.
2017-06-24 01:40:02 +08:00
assert(info->state == background_thread_paused);
Move background thread creation to background_thread_0. To avoid complications, avoid invoking pthread_create "internally", instead rely on thread0 to launch new threads, and also terminating threads when asked.
2017-06-09 13:46:31 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &info->mtx);
Combine background_thread started / paused into state.
2017-06-10 06:45:25 +08:00
info->state = background_thread_started;
Move background thread creation to background_thread_0. To avoid complications, avoid invoking pthread_create "internally", instead rely on thread0 to launch new threads, and also terminating threads when asked.
2017-06-09 13:46:31 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &info->mtx);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
}
malloc_mutex_unlock(tsd_tsdn(tsd), &background_thread_lock);
}
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
static int
arena_i_reset_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
int ret;
unsigned arena_ind;
arena_t *arena;
ret = arena_i_reset_destroy_helper(tsd, mib, miblen, oldp, oldlenp,
newp, newlen, &arena_ind, &arena);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (ret != 0) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_reset_prepare_background_thread(tsd, arena_ind);
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
arena_reset(tsd, arena);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_reset_finish_background_thread(tsd, arena_ind);
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
}
static int
arena_i_destroy_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
int ret;
unsigned arena_ind;
arena_t *arena;
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena_t *ctl_darena, *ctl_arena;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
ret = arena_i_reset_destroy_helper(tsd, mib, miblen, oldp, oldlenp,
newp, newlen, &arena_ind, &arena);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (ret != 0) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
goto label_return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
if (arena_nthreads_get(arena, false) != 0 || arena_nthreads_get(arena,
true) != 0) {
ret = EFAULT;
goto label_return;
}
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_reset_prepare_background_thread(tsd, arena_ind);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
/* Merge stats after resetting and purging arena. */
arena_reset(tsd, arena);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_decay(tsd_tsdn(tsd), arena, false, true);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_darena = arenas_i(MALLCTL_ARENAS_DESTROYED);
ctl_darena->initialized = true;
ctl_arena_refresh(tsd_tsdn(tsd), arena, ctl_darena, arena_ind, true);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
/* Destroy arena. */
arena_destroy(tsd, arena);
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arena = arenas_i(arena_ind);
ctl_arena->initialized = false;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
/* Record arena index for later recycling via arenas.create. */
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ql_elm_new(ctl_arena, destroyed_link);
ql_tail_insert(&ctl_arenas->destroyed, ctl_arena, destroyed_link);
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_reset_finish_background_thread(tsd, arena_ind);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
assert(ret == 0);
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static int
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
arena_i_dss_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t *oldlenp, void *newp, size_t newlen) {
Fix arena.<i>.dss mallctl to handle read-only calls.
2014-08-16 03:20:20 +08:00
int ret;
const char *dss = NULL;
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
unsigned arena_ind;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
dss_prec_t dss_prec_old = dss_prec_limit;
dss_prec_t dss_prec = dss_prec_limit;
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
WRITE(dss, const char *);
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
MIB_UNSIGNED(arena_ind, 1);
Fix arena.<i>.dss mallctl to handle read-only calls.
2014-08-16 03:20:20 +08:00
if (dss != NULL) {
int i;
bool match = false;
for (i = 0; i < dss_prec_limit; i++) {
if (strcmp(dss_prec_names[i], dss) == 0) {
dss_prec = i;
match = true;
break;
}
}
Convert to uniform style: cond == false --> !cond
2014-10-04 01:16:09 +08:00
if (!match) {
Fix arena.<i>.dss mallctl to handle read-only calls.
2014-08-16 03:20:20 +08:00
ret = EINVAL;
goto label_return;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
}
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
/*
* Access via index narenas is deprecated, and scheduled for removal in
* 6.0.0.
*/
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
if (arena_ind == MALLCTL_ARENAS_ALL || arena_ind ==
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
ctl_arenas->narenas) {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
if (dss_prec != dss_prec_limit &&
extent_dss_prec_set(dss_prec)) {
Fix arena.<i>.dss mallctl to handle read-only calls.
2014-08-16 03:20:20 +08:00
ret = EFAULT;
goto label_return;
}
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
dss_prec_old = extent_dss_prec_get();
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
} else {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
arena_t *arena = arena_get(tsd_tsdn(tsd), arena_ind, false);
if (arena == NULL || (dss_prec != dss_prec_limit &&
Convert arena->dss_prec synchronization to atomics.
2017-02-13 08:34:36 +08:00
arena_dss_prec_set(arena, dss_prec))) {
Fix arena.<i>.dss mallctl to handle read-only calls.
2014-08-16 03:20:20 +08:00
ret = EFAULT;
goto label_return;
}
Convert arena->dss_prec synchronization to atomics.
2017-02-13 08:34:36 +08:00
dss_prec_old = arena_dss_prec_get(arena);
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
Fix arena.<i>.dss mallctl to handle read-only calls.
2014-08-16 03:20:20 +08:00
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
dss = dss_prec_names[dss_prec_old];
READ(dss, const char *);
ret = 0;
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
static int
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
arena_i_decay_ms_ctl_impl(tsd_t *tsd, const size_t *mib, size_t miblen,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen, bool dirty) {
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
int ret;
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
unsigned arena_ind;
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
arena_t *arena;
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
MIB_UNSIGNED(arena_ind, 1);
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
arena = arena_get(tsd_tsdn(tsd), arena_ind, false);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
if (arena == NULL) {
ret = EFAULT;
goto label_return;
}
if (oldp != NULL && oldlenp != NULL) {
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
size_t oldval = dirty ? arena_dirty_decay_ms_get(arena) :
arena_muzzy_decay_ms_get(arena);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
READ(oldval, ssize_t);
}
if (newp != NULL) {
if (newlen != sizeof(ssize_t)) {
ret = EINVAL;
goto label_return;
}
Avoid creating bg thds for huge arena lone. For low arena count settings, the huge threshold feature may trigger an unwanted bg thd creation. Given that the huge arena does eager purging by default, bypass bg thd creation when initializing the huge arena.
2019-01-15 06:16:09 +08:00
if (arena_is_huge(arena_ind) && *(ssize_t *)newp > 0) {
/*
* By default the huge arena purges eagerly. If it is
* set to non-zero decay time afterwards, background
* thread might be needed.
*/
if (background_thread_create(tsd, arena_ind)) {
ret = EFAULT;
goto label_return;
}
}
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
if (dirty ? arena_dirty_decay_ms_set(tsd_tsdn(tsd), arena,
*(ssize_t *)newp) : arena_muzzy_decay_ms_set(tsd_tsdn(tsd),
arena, *(ssize_t *)newp)) {
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
ret = EFAULT;
goto label_return;
}
}
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
}
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
static int
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
arena_i_dirty_decay_ms_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
return arena_i_decay_ms_ctl_impl(tsd, mib, miblen, oldp, oldlenp, newp,
newlen, true);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
}
static int
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
arena_i_muzzy_decay_ms_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
return arena_i_decay_ms_ctl_impl(tsd, mib, miblen, oldp, oldlenp, newp,
newlen, false);
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
static int
s/chunk_hook/extent_hook/g
2016-06-01 06:03:51 +08:00
arena_i_extent_hooks_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
int ret;
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
unsigned arena_ind;
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
arena_t *arena;
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
Range-check mib[1] --> arena_ind casts.
2017-01-04 23:51:49 +08:00
MIB_UNSIGNED(arena_ind, 1);
Allow setting extent hooks on uninitialized auto arenas. Setting extent hooks can result in initializing an unused auto arena. This is useful to install extent hooks on auto arenas from the beginning.
2018-04-12 06:15:26 +08:00
if (arena_ind < narenas_total_get()) {
extent_hooks_t *old_extent_hooks;
arena = arena_get(tsd_tsdn(tsd), arena_ind, false);
if (arena == NULL) {
if (arena_ind >= narenas_auto) {
ret = EFAULT;
goto label_return;
}
old_extent_hooks =
Extent -> Ehooks: Move over default hooks.
2019-12-04 06:23:40 +08:00
(extent_hooks_t *)&ehooks_default_extent_hooks;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
READ(old_extent_hooks, extent_hooks_t *);
Allow setting extent hooks on uninitialized auto arenas. Setting extent hooks can result in initializing an unused auto arena. This is useful to install extent hooks on auto arenas from the beginning.
2018-04-12 06:15:26 +08:00
if (newp != NULL) {
/* Initialize a new arena as a side effect. */
extent_hooks_t *new_extent_hooks
JEMALLOC_CC_SILENCE_INIT(NULL);
WRITE(new_extent_hooks, extent_hooks_t *);
arena = arena_init(tsd_tsdn(tsd), arena_ind,
new_extent_hooks);
if (arena == NULL) {
ret = EFAULT;
goto label_return;
}
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
} else {
Allow setting extent hooks on uninitialized auto arenas. Setting extent hooks can result in initializing an unused auto arena. This is useful to install extent hooks on auto arenas from the beginning.
2018-04-12 06:15:26 +08:00
if (newp != NULL) {
extent_hooks_t *new_extent_hooks
JEMALLOC_CC_SILENCE_INIT(NULL);
WRITE(new_extent_hooks, extent_hooks_t *);
Move extent hook getters/setters to arena.c This is where they're logically scoped; they access arena data.
2019-11-19 04:59:34 +08:00
old_extent_hooks = arena_set_extent_hooks(tsd,
arena, new_extent_hooks);
Allow setting extent hooks on uninitialized auto arenas. Setting extent hooks can result in initializing an unused auto arena. This is useful to install extent hooks on auto arenas from the beginning.
2018-04-12 06:15:26 +08:00
READ(old_extent_hooks, extent_hooks_t *);
} else {
Move extent ehook calls into ehooks
2019-12-03 06:19:22 +08:00
old_extent_hooks =
ehooks_get_extent_hooks_ptr(
arena_get_ehooks(arena));
Allow setting extent hooks on uninitialized auto arenas. Setting extent hooks can result in initializing an unused auto arena. This is useful to install extent hooks on auto arenas from the beginning.
2018-04-12 06:15:26 +08:00
READ(old_extent_hooks, extent_hooks_t *);
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
}
} else {
ret = EFAULT;
goto label_return;
}
ret = 0;
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement dynamic per arena control over dirty page purging. Add mallctls: - arenas.lg_dirty_mult is initialized via opt.lg_dirty_mult, and can be modified to change the initial lg_dirty_mult setting for newly created arenas. - arena.<i>.lg_dirty_mult controls an individual arena's dirty page purging threshold, and synchronously triggers any purging that may be necessary to maintain the constraint. - arena.<i>.chunk.purge allows the per arena dirty page purging function to be replaced. This resolves #93.
2015-03-19 09:55:33 +08:00
}
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arena_i_retain_grow_limit_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
int ret;
unsigned arena_ind;
arena_t *arena;
if (!opt_retain) {
/* Only relevant when retain is enabled. */
return ENOENT;
}
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
MIB_UNSIGNED(arena_ind, 1);
if (arena_ind < narenas_total_get() && (arena =
arena_get(tsd_tsdn(tsd), arena_ind, false)) != NULL) {
size_t old_limit, new_limit;
if (newp != NULL) {
WRITE(new_limit, size_t);
}
bool err = arena_retain_grow_limit_get_set(tsd, arena,
&old_limit, newp != NULL ? &new_limit : NULL);
if (!err) {
READ(old_limit, size_t);
ret = 0;
} else {
ret = EFAULT;
}
} else {
ret = EFAULT;
}
label_return:
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
return ret;
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static const ctl_named_node_t *
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arena_i_index(tsdn_t *tsdn, const size_t *mib, size_t miblen,
size_t i) {
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
const ctl_named_node_t *ret;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsdn, &ctl_mtx);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
switch (i) {
case MALLCTL_ARENAS_ALL:
case MALLCTL_ARENAS_DESTROYED:
break;
default:
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
if (i > ctl_arenas->narenas) {
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
ret = NULL;
goto label_return;
}
break;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
ret = super_arena_i_node;
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsdn, &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/******************************************************************************/
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arenas_narenas_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
int ret;
unsigned narenas;
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
READONLY();
if (*oldlenp != sizeof(unsigned)) {
ret = EINVAL;
goto label_return;
}
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
narenas = ctl_arenas->narenas;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
READ(narenas, unsigned);
ret = 0;
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arenas_decay_ms_ctl_impl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen, bool dirty) {
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
int ret;
if (oldp != NULL && oldlenp != NULL) {
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
size_t oldval = (dirty ? arena_dirty_decay_ms_default_get() :
arena_muzzy_decay_ms_default_get());
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
READ(oldval, ssize_t);
}
if (newp != NULL) {
if (newlen != sizeof(ssize_t)) {
ret = EINVAL;
goto label_return;
}
Add arena.i.retain_grow_limit This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
2017-11-03 08:48:39 +08:00
if (dirty ? arena_dirty_decay_ms_default_set(*(ssize_t *)newp)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
: arena_muzzy_decay_ms_default_set(*(ssize_t *)newp)) {
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
ret = EFAULT;
goto label_return;
}
}
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
}
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
static int
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
arenas_dirty_decay_ms_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
return arenas_decay_ms_ctl_impl(tsd, mib, miblen, oldp, oldlenp, newp,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
newlen, true);
}
static int
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
arenas_muzzy_decay_ms_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
return arenas_decay_ms_ctl_impl(tsd, mib, miblen, oldp, oldlenp, newp,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
newlen, false);
}
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
CTL_RO_NL_GEN(arenas_quantum, QUANTUM, size_t)
Clean up *PAGE* macros. s/PAGE_SHIFT/LG_PAGE/g and s/PAGE_SIZE/PAGE/g. Remove remnants of the dynamic-page-shift code. Rename the "arenas.pagesize" mallctl to "arenas.page". Remove the "arenas.chunksize" mallctl, which is redundant with "opt.lg_chunk".
2012-04-02 22:04:34 +08:00
CTL_RO_NL_GEN(arenas_page, PAGE, size_t)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_NL_GEN(arenas_tcache_max, tcache_maxclass, size_t)
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
CTL_RO_NL_GEN(arenas_nbins, SC_NBINS, unsigned)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_NL_GEN(arenas_nhbins, nhbins, unsigned)
Pull out arena_bin_info_t and arena_bin_t into their own file. In the process, kill arena_bin_index, which is unused. To follow are several diffs continuing this separation.
2017-10-02 08:22:06 +08:00
CTL_RO_NL_GEN(arenas_bin_i_size, bin_infos[mib[2]].reg_size, size_t)
CTL_RO_NL_GEN(arenas_bin_i_nregs, bin_infos[mib[2]].nregs, uint32_t)
CTL_RO_NL_GEN(arenas_bin_i_slab_size, bin_infos[mib[2]].slab_size, size_t)
Add stats for arenas.bin.i.nshards.
2018-11-22 03:17:31 +08:00
CTL_RO_NL_GEN(arenas_bin_i_nshards, bin_infos[mib[2]].n_shards, uint32_t)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
static const ctl_named_node_t *
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arenas_bin_i_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t i) {
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
if (i > SC_NBINS) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return super_arenas_bin_i_node;
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
}
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
CTL_RO_NL_GEN(arenas_nlextents, SC_NSIZES - SC_NBINS, unsigned)
CTL_RO_NL_GEN(arenas_lextent_i_size, sz_index2size(SC_NBINS+(szind_t)mib[2]),
Header refactoring: Pull size helpers out of jemalloc module.
2017-05-31 01:45:37 +08:00
size_t)
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
static const ctl_named_node_t *
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arenas_lextent_i_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t i) {
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
if (i > SC_NSIZES - SC_NBINS) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return super_arenas_lextent_i_node;
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arenas_create_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
int ret;
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
extent_hooks_t *extent_hooks;
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
unsigned arena_ind;
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
Extent -> Ehooks: Move over default hooks.
2019-12-04 06:23:40 +08:00
extent_hooks = (extent_hooks_t *)&ehooks_default_extent_hooks;
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
WRITE(extent_hooks, extent_hooks_t *);
Switch ctl to explicitly use tsd instead of tsdn.
2017-06-23 09:58:40 +08:00
if ((arena_ind = ctl_arena_init(tsd, extent_hooks)) == UINT_MAX) {
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
ret = EAGAIN;
goto label_return;
Add the "arenas.purge" mallctl.
2010-10-01 07:55:08 +08:00
}
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
READ(arena_ind, unsigned);
Add the "arenas.purge" mallctl.
2010-10-01 07:55:08 +08:00
ret = 0;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add the "arenas.purge" mallctl.
2010-10-01 07:55:08 +08:00
}
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
arenas_lookup_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
int ret;
unsigned arena_ind;
void *ptr;
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
edata_t *edata;
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
arena_t *arena;
ptr = NULL;
ret = EINVAL;
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
WRITE(ptr, void *);
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
edata = emap_edata_lookup(tsd_tsdn(tsd), &emap_global, ptr);
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
if (edata == NULL)
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
goto label_return;
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
arena = arena_get_from_edata(edata);
Mallctl: Add arenas.lookup Implement a new mallctl operation that allows looking up the arena a region of memory belongs to.
2018-05-02 02:31:09 +08:00
if (arena == NULL)
goto label_return;
arena_ind = arena_ind_get(arena);
READ(arena_ind, unsigned);
ret = 0;
label_return:
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
return ret;
}
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
/******************************************************************************/
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
prof_thread_active_init_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp,
size_t newlen) {
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
int ret;
bool oldval;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
if (newp != NULL) {
if (newlen != sizeof(bool)) {
ret = EINVAL;
goto label_return;
}
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
oldval = prof_thread_active_init_set(tsd_tsdn(tsd),
*(bool *)newp);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
oldval = prof_thread_active_init_get(tsd_tsdn(tsd));
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
READ(oldval, bool);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
}
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
prof_active_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
int ret;
bool oldval;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Guard prof_active reset by opt_prof Set `prof_active` to read-only when `opt_prof` is turned off.
2019-08-28 05:42:14 +08:00
ret = ENOENT;
goto label_return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
if (newp != NULL) {
Implement/test/fix prof-related mallctl's. Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
2014-10-04 14:25:30 +08:00
if (newlen != sizeof(bool)) {
ret = EINVAL;
goto label_return;
}
Guard prof_active reset by opt_prof Set `prof_active` to read-only when `opt_prof` is turned off.
2019-08-28 05:42:14 +08:00
bool val = *(bool *)newp;
if (!opt_prof && val) {
ret = ENOENT;
goto label_return;
}
oldval = prof_active_set(tsd_tsdn(tsd), val);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
oldval = prof_active_get(tsd_tsdn(tsd));
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
READ(oldval, bool);
ret = 0;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Add sampling activation/deactivation control. Add the E/e options to control whether the application starts with sampling active/inactive (secondary control to F/f). Add the prof.active mallctl so that the application can activate/deactivate sampling on the fly.
2010-04-01 09:43:24 +08:00
}
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
prof_dump_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
int ret;
Allow prof.dump mallctl to specify filename.
2010-03-03 03:57:30 +08:00
const char *filename = NULL;
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
Allow prof.dump mallctl to specify filename.
2010-03-03 03:57:30 +08:00
WRITEONLY();
WRITE(filename, const char *);
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
Add witness, a simple online locking validator. This resolves #358.
2016-04-14 14:36:15 +08:00
if (prof_mdump(tsd, filename)) {
Allow prof.dump mallctl to specify filename.
2010-03-03 03:57:30 +08:00
ret = EFAULT;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Allow prof.dump mallctl to specify filename.
2010-03-03 03:57:30 +08:00
}
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
ret = 0;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
}
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
prof_gdump_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
int ret;
bool oldval;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
if (newp != NULL) {
if (newlen != sizeof(bool)) {
ret = EINVAL;
goto label_return;
}
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
oldval = prof_gdump_set(tsd_tsdn(tsd), *(bool *)newp);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
} else {
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
oldval = prof_gdump_get(tsd_tsdn(tsd));
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
READ(oldval, bool);
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement the prof.gdump mallctl. This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
2015-01-26 13:16:57 +08:00
}
Add "prof.dump_prefix" to override filename prefixes for dumps.
2019-09-10 11:18:41 +08:00
static int
prof_dump_prefix_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
const char *prefix = NULL;
if (!config_prof) {
return ENOENT;
}
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
WRITEONLY();
WRITE(prefix, const char *);
ret = prof_dump_prefix_set(tsd_tsdn(tsd), prefix) ? EFAULT : 0;
label_return:
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
return ret;
}
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
prof_reset_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
int ret;
size_t lg_sample = lg_prof_sample;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (!config_prof) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ENOENT;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
WRITEONLY();
WRITE(lg_sample, size_t);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (lg_sample >= (sizeof(uint64_t) << 3)) {
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
lg_sample = (sizeof(uint64_t) << 3) - 1;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
Do not (recursively) allocate within tsd_fetch(). Refactor tsd so that tsdn_fetch() does not trigger allocation, since allocation could cause infinite recursion. This resolves #458.
2016-10-21 14:59:12 +08:00
prof_reset(tsd, lg_sample);
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
ret = 0;
label_return:
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
}
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_NL_CGEN(config_prof, prof_interval, prof_interval, uint64_t)
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
CTL_RO_NL_CGEN(config_prof, lg_prof_sample, lg_prof_sample, size_t)
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
static int
prof_log_start_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
size_t *oldlenp, void *newp, size_t newlen) {
int ret;
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
const char *filename = NULL;
if (!config_prof) {
return ENOENT;
}
WRITEONLY();
WRITE(filename, const char *);
if (prof_log_start(tsd_tsdn(tsd), filename)) {
ret = EFAULT;
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
goto label_return;
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
}
ret = 0;
label_return:
return ret;
}
static int
prof_log_stop_ctl(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
size_t *oldlenp, void *newp, size_t newlen) {
if (!config_prof) {
return ENOENT;
}
if (prof_log_stop(tsd_tsdn(tsd))) {
return EFAULT;
}
return 0;
}
Implement interval-based heap profile dumping. Add mallctl interfaces for profiling parameters. Fix a file descriptor leak in heap profile dumping.
2010-02-12 05:19:21 +08:00
/******************************************************************************/
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
CTL_RO_CGEN(config_stats, stats_allocated, ctl_stats->allocated, size_t)
CTL_RO_CGEN(config_stats, stats_active, ctl_stats->active, size_t)
CTL_RO_CGEN(config_stats, stats_metadata, ctl_stats->metadata, size_t)
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
CTL_RO_CGEN(config_stats, stats_metadata_thp, ctl_stats->metadata_thp, size_t)
Refactor ctl_stats_t. Refactor ctl_stats_t to be a demand-zeroed non-growing data structure. To keep the size from being onerous (~60 MiB) on 32-bit systems, convert the arenas field to contain pointers rather than directly embedded ctl_arena_stats_t elements.
2017-01-04 04:40:54 +08:00
CTL_RO_CGEN(config_stats, stats_resident, ctl_stats->resident, size_t)
CTL_RO_CGEN(config_stats, stats_mapped, ctl_stats->mapped, size_t)
CTL_RO_CGEN(config_stats, stats_retained, ctl_stats->retained, size_t)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
Add background thread related stats.
2017-05-13 03:30:33 +08:00
CTL_RO_CGEN(config_stats, stats_background_thread_num_threads,
ctl_stats->background_thread.num_threads, size_t)
CTL_RO_CGEN(config_stats, stats_background_thread_num_runs,
ctl_stats->background_thread.num_runs, uint64_t)
CTL_RO_CGEN(config_stats, stats_background_thread_run_interval,
nstime_ns(&ctl_stats->background_thread.run_interval), uint64_t)
Add stats counters for number of zero reallocs
2019-10-27 02:04:46 +08:00
CTL_RO_CGEN(config_stats, stats_zero_reallocs,
atomic_load_zu(&zero_realloc_count, ATOMIC_RELAXED), size_t)
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
CTL_RO_GEN(stats_arenas_i_dss, arenas_i(mib[2])->dss, const char *)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_RO_GEN(stats_arenas_i_dirty_decay_ms, arenas_i(mib[2])->dirty_decay_ms,
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
ssize_t)
Refactor *decay_time into *decay_ms. Support millisecond resolution for decay times. Among other use cases this makes it possible to specify a short initial dirty-->muzzy decay phase, followed by a longer muzzy-->clean decay phase. This resolves #812.
2017-05-18 01:47:00 +08:00
CTL_RO_GEN(stats_arenas_i_muzzy_decay_ms, arenas_i(mib[2])->muzzy_decay_ms,
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
ssize_t)
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
CTL_RO_GEN(stats_arenas_i_nthreads, arenas_i(mib[2])->nthreads, unsigned)
Add stats: arena uptime.
2017-05-17 04:56:00 +08:00
CTL_RO_GEN(stats_arenas_i_uptime,
nstime_ns(&arenas_i(mib[2])->astats->astats.uptime), uint64_t)
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
CTL_RO_GEN(stats_arenas_i_pactive, arenas_i(mib[2])->pactive, size_t)
CTL_RO_GEN(stats_arenas_i_pdirty, arenas_i(mib[2])->pdirty, size_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_GEN(stats_arenas_i_pmuzzy, arenas_i(mib[2])->pmuzzy, size_t)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_mapped,
PA: Move mapped into pa stats.
2020-03-10 03:06:19 +08:00
locked_read_atomic_zu(&arenas_i(
mib[2])->astats->astats.pa_shard_stats.mapped), size_t)
Add the stats.retained and stats.arenas.<i>.retained statistics. This resolves #367.
2016-05-04 13:11:35 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_retained,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_atomic_zu(&arenas_i(mib[2])->astats->astats.retained),
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
size_t)
Add stats for the size of extent_avail heap
2018-08-02 05:14:33 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_extent_avail,
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
atomic_load_zu(&arenas_i(mib[2])->astats->astats.edata_avail,
Add stats for the size of extent_avail heap
2018-08-02 05:14:33 +08:00
ATOMIC_RELAXED),
size_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_dirty_npurge,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
&arenas_i(mib[2])->astats->astats.pa_shard_stats.decay_dirty.npurge),
uint64_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_dirty_nmadvise,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
&arenas_i(mib[2])->astats->astats.pa_shard_stats.decay_dirty.nmadvise),
uint64_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_dirty_purged,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
&arenas_i(mib[2])->astats->astats.pa_shard_stats.decay_dirty.purged),
uint64_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_muzzy_npurge,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
&arenas_i(mib[2])->astats->astats.pa_shard_stats.decay_muzzy.npurge),
uint64_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_muzzy_nmadvise,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
&arenas_i(mib[2])->astats->astats.pa_shard_stats.decay_muzzy.nmadvise),
uint64_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_muzzy_purged,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
PA: Move in decay stats.
2020-03-10 01:40:37 +08:00
&arenas_i(mib[2])->astats->astats.pa_shard_stats.decay_muzzy.purged),
uint64_t)
Implement two-phase decay-based purging. Split decay-based purging into two phases, the first of which uses lazy purging to convert dirty pages to "muzzy", and the second of which uses forced purging, decommit, or unmapping to convert pages to clean or destroy them altogether. Not all operating systems support lazy purging, yet the application may provide extent hooks that implement lazy purging, so care must be taken to dynamically omit the first phase when necessary. The mallctl interfaces change as follows: - opt.decay_time --> opt.{dirty,muzzy}_decay_time - arena.<i>.decay_time --> arena.<i>.{dirty,muzzy}_decay_time - arenas.decay_time --> arenas.{dirty,muzzy}_decay_time - stats.arenas.<i>.pdirty --> stats.arenas.<i>.p{dirty,muzzy} - stats.arenas.<i>.{npurge,nmadvise,purged} --> stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged} This resolves #521.
2017-03-09 14:42:57 +08:00
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_base,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
atomic_load_zu(&arenas_i(mib[2])->astats->astats.base, ATOMIC_RELAXED),
size_t)
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_internal,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
atomic_load_zu(&arenas_i(mib[2])->astats->astats.internal, ATOMIC_RELAXED),
size_t)
Add stats for metadata_thp. Report number of THPs used in arena and aggregated stats.
2017-08-26 04:24:49 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_metadata_thp,
atomic_load_zu(&arenas_i(mib[2])->astats->astats.metadata_thp,
ATOMIC_RELAXED), size_t)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_tcache_bytes,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
atomic_load_zu(&arenas_i(mib[2])->astats->astats.tcache_bytes,
ATOMIC_RELAXED), size_t)
Implement per arena base allocators. Add/rename related mallctls: - Add stats.arenas.<i>.base . - Rename stats.arenas.<i>.metadata to stats.arenas.<i>.internal . - Add stats.arenas.<i>.resident . Modify the arenas.extend mallctl to take an optional (extent_hooks_t *) argument so that it is possible for all base allocations to be serviced by the specified extent hooks. This resolves #463.
2016-12-23 06:39:10 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_resident,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
atomic_load_zu(&arenas_i(mib[2])->astats->astats.resident, ATOMIC_RELAXED),
size_t)
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_abandoned_vm,
PA: Add a stats type.
2020-03-09 04:08:15 +08:00
atomic_load_zu(
&arenas_i(mib[2])->astats->astats.pa_shard_stats.abandoned_vm,
Track the leaked VM space via the abandoned_vm counter. The counter is 0 unless metadata allocation failed (indicates OOM), and is mainly for sanity checking.
2019-07-16 06:56:05 +08:00
ATOMIC_RELAXED), size_t)
Fix a few mallctl() documentation errors. Normalize mallctl() order (code and documentation).
2013-12-20 13:40:41 +08:00
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_small_allocated,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arenas_i(mib[2])->astats->allocated_small, size_t)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_small_nmalloc,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arenas_i(mib[2])->astats->nmalloc_small, uint64_t)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_small_ndalloc,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arenas_i(mib[2])->astats->ndalloc_small, uint64_t)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_small_nrequests,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arenas_i(mib[2])->astats->nrequests_small, uint64_t)
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_small_nfills,
arenas_i(mib[2])->astats->nfills_small, uint64_t)
CTL_RO_CGEN(config_stats, stats_arenas_i_small_nflushes,
arenas_i(mib[2])->astats->nflushes_small, uint64_t)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_large_allocated,
Convert remaining arena_stats_t fields to atomics These were all size_ts, so we have atomics support for them on all platforms, so the conversion is straightforward. Left non-atomic is curlextents, which AFAICT is not used atomically anywhere.
2017-03-14 07:18:40 +08:00
atomic_load_zu(&arenas_i(mib[2])->astats->astats.allocated_large,
ATOMIC_RELAXED), size_t)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_large_nmalloc,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
&arenas_i(mib[2])->astats->astats.nmalloc_large), uint64_t)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_large_ndalloc,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
&arenas_i(mib[2])->astats->astats.ndalloc_large), uint64_t)
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_large_nrequests,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
&arenas_i(mib[2])->astats->astats.nrequests_large), uint64_t)
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
/*
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
* Note: "nmalloc_large" here instead of "nfills" in the read. This is
* intentional (large has no batch fill).
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
*/
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_large_nfills,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
&arenas_i(mib[2])->astats->astats.nmalloc_large), uint64_t)
CTL_RO_CGEN(config_stats, stats_arenas_i_large_nflushes,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Track nfills and nflushes for arenas.i.small / large. Small is added purely for convenience. Large flushes wasn't tracked before and can be useful in analysis. Large fill simply reports nmalloc, since there is no batch fill for large currently.
2019-05-07 07:36:55 +08:00
&arenas_i(mib[2])->astats->astats.nflushes_large), uint64_t)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
/* Lock profiling related APIs below. */
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
#define RO_MUTEX_CTL_GEN(n, l) \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL_RO_CGEN(config_stats, stats_##n##_num_ops, \
l.n_lock_ops, uint64_t) \
CTL_RO_CGEN(config_stats, stats_##n##_num_wait, \
l.n_wait_times, uint64_t) \
CTL_RO_CGEN(config_stats, stats_##n##_num_spin_acq, \
l.n_spin_acquired, uint64_t) \
CTL_RO_CGEN(config_stats, stats_##n##_num_owner_switch, \
l.n_owner_switches, uint64_t) \
CTL_RO_CGEN(config_stats, stats_##n##_total_wait_time, \
Switch to nstime_t for the time related fields in mutex profiling.
2017-03-18 08:42:10 +08:00
nstime_ns(&l.tot_wait_time), uint64_t) \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL_RO_CGEN(config_stats, stats_##n##_max_wait_time, \
Switch to nstime_t for the time related fields in mutex profiling.
2017-03-18 08:42:10 +08:00
nstime_ns(&l.max_wait_time), uint64_t) \
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
CTL_RO_CGEN(config_stats, stats_##n##_max_num_thds, \
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
l.max_n_thds, uint32_t)
Added lock profiling and output for global locks (ctl, prof and base).
2017-03-12 12:28:31 +08:00
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
/* Global mutexes. */
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#define OP(mtx) \
RO_MUTEX_CTL_GEN(mutexes_##mtx, \
ctl_stats->mutex_prof_data[global_prof_mutex_##mtx])
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_GLOBAL_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
/* Per arena mutexes */
#define OP(mtx) RO_MUTEX_CTL_GEN(arenas_i_mutexes_##mtx, \
arenas_i(mib[2])->astats->astats.mutex_prof_data[arena_prof_mutex_##mtx])
Header refactoring: ctl - unify and remove from catchall. In order to do this, we introduce the mutex_prof module, which breaks a circular dependency between ctl and prof.
2017-04-25 08:09:56 +08:00
MUTEX_PROF_ARENA_MUTEXES
Refactor mutex profiling code with x-macros.
2017-03-22 02:56:38 +08:00
#undef OP
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
Added extents_dirty / _muzzy mutexes, as well as decay_dirty / _muzzy.
2017-03-16 05:00:57 +08:00
/* tcache bin mutex */
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
RO_MUTEX_CTL_GEN(arenas_i_bins_j_mutex,
arenas_i(mib[2])->astats->bstats[mib[4]].mutex_data)
#undef RO_MUTEX_CTL_GEN
/* Resets all mutex stats, including global, arena and bin mutexes. */
static int
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
stats_mutexes_reset_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp,
void *newp, size_t newlen) {
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
if (!config_stats) {
return ENOENT;
}
tsdn_t *tsdn = tsd_tsdn(tsd);
#define MUTEX_PROF_RESET(mtx) \
malloc_mutex_lock(tsdn, &mtx); \
malloc_mutex_prof_data_reset(tsdn, &mtx); \
malloc_mutex_unlock(tsdn, &mtx);
Profile per arena base mutex, instead of just a0.
2017-03-22 16:49:56 +08:00
/* Global mutexes: ctl and prof. */
MUTEX_PROF_RESET(ctl_mtx);
Add profiling for the background thread mutex.
2017-05-13 07:26:59 +08:00
if (have_background_thread) {
MUTEX_PROF_RESET(background_thread_lock);
}
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
if (config_prof && opt_prof) {
MUTEX_PROF_RESET(bt2gctx_mtx);
Report stats for tdatas_mtx and prof_dump_mtx
2019-08-03 00:41:35 +08:00
MUTEX_PROF_RESET(tdatas_mtx);
MUTEX_PROF_RESET(prof_dump_mtx);
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
}
Profile per arena base mutex, instead of just a0.
2017-03-22 16:49:56 +08:00
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
/* Per arena mutexes. */
unsigned n = narenas_total_get();
for (unsigned i = 0; i < n; i++) {
arena_t *arena = arena_get(tsdn, i, false);
if (!arena) {
continue;
}
MUTEX_PROF_RESET(arena->large_mtx);
PA: Move in the arena edata_cache.
2020-03-09 02:41:19 +08:00
MUTEX_PROF_RESET(arena->pa_shard.edata_cache.mtx);
PA: Move in the ecache_t objects.
2020-03-09 01:35:56 +08:00
MUTEX_PROF_RESET(arena->pa_shard.ecache_dirty.mtx);
MUTEX_PROF_RESET(arena->pa_shard.ecache_muzzy.mtx);
MUTEX_PROF_RESET(arena->pa_shard.ecache_retained.mtx);
Introduce decay module and put decay objects in PA
2020-03-10 04:11:35 +08:00
MUTEX_PROF_RESET(arena->pa_shard.decay_dirty.mtx);
MUTEX_PROF_RESET(arena->pa_shard.decay_muzzy.mtx);
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
MUTEX_PROF_RESET(arena->tcache_ql_mtx);
Profile per arena base mutex, instead of just a0.
2017-03-22 16:49:56 +08:00
MUTEX_PROF_RESET(arena->base->mtx);
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
for (szind_t i = 0; i < SC_NBINS; i++) {
Add support for sharded bins within an arena. This makes it possible to have multiple set of bins in an arena, which improves arena scalability because the bins (especially the small ones) are always the limiting factor in production workload. A bin shard is picked on allocation; each extent tracks the bin shard id for deallocation. The shard size will be determined using runtime options.
2018-11-13 07:56:04 +08:00
for (unsigned j = 0; j < bin_infos[i].n_shards; j++) {
bin_t *bin = &arena->bins[i].bin_shards[j];
MUTEX_PROF_RESET(bin->lock);
}
Added "stats.mutexes.reset" mallctl to reset all mutex stats. Also switched from the term "lock" to "mutex".
2017-03-14 08:29:03 +08:00
}
}
#undef MUTEX_PROF_RESET
return 0;
}
Add arena lock stats output.
2017-03-11 04:14:05 +08:00
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nmalloc,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.nmalloc, uint64_t)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_ndalloc,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.ndalloc, uint64_t)
Reduce cpp conditional logic complexity. Convert configuration-related cpp conditional logic to use static constant variables, e.g.: #ifdef JEMALLOC_DEBUG [...] #endif becomes: if (config_debug) { [...] } The advantage is clearer, more concise code. The main disadvantage is that data structures no longer have conditionally defined fields, so they pay the cost of all fields regardless of whether they are used. In practice, this is only a minor concern; config_stats will go away in an upcoming change, and config_prof is the only other major feature that depends on more than a few special-purpose fields.
2012-02-11 12:22:09 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nrequests,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.nrequests, uint64_t)
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_curregs,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.curregs, size_t)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nfills,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.nfills, uint64_t)
Remove --disable-tcache. Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
2017-04-21 08:21:37 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nflushes,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.nflushes, uint64_t)
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nslabs,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.nslabs, uint64_t)
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nreslabs,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.reslabs, uint64_t)
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_curslabs,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.curslabs, size_t)
Add nonfull_slabs to bin_stats_t. When config_stats is enabled track the size of bin->slabs_nonfull in the new nonfull_slabs counter in bin_stats_t. This metric should be useful for establishing an upper ceiling on the savings possible by meshing.
2019-04-12 19:08:50 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_bins_j_nonfull_slabs,
Reduce footprint of bin_t. Avoid storing mutex_prof_data_t in bin_t. Added bin_stats_data_t which is used for reporting bin stats.
2019-11-06 12:43:59 +08:00
arenas_i(mib[2])->astats->bstats[mib[4]].stats_data.nonfull_slabs, size_t)
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static const ctl_named_node_t *
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
stats_arenas_i_bins_j_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t j) {
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
if (j > SC_NBINS) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return super_stats_arenas_i_bins_j_node;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_lextents_j_nmalloc,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
&arenas_i(mib[2])->astats->lstats[mib[4]].nmalloc), uint64_t)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_lextents_j_ndalloc,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
&arenas_i(mib[2])->astats->lstats[mib[4]].ndalloc), uint64_t)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_lextents_j_nrequests,
Introduce lockedint module. This pulls out the various abstractions where some stats counter is sometimes an atomic, sometimes a plain variable, sometimes always protected by a lock, sometimes protected by reads but not writes, etc. With this change, these cases are treated consistently, and access patterns tagged. In the process, we fix a few missed-update bugs (where one caller assumes "protected-by-a-lock" semantics and another does not).
2020-03-09 11:43:41 +08:00
locked_read_u64_unsynchronized(
Split up and standardize naming of stats code. The arena-associated stats are now all prefixed with arena_stats_, and live in their own file. Likewise, malloc_bin_stats_t -> bin_stats_t, also in its own file.
2017-11-05 03:50:19 +08:00
&arenas_i(mib[2])->astats->lstats[mib[4]].nrequests), uint64_t)
Rename huge to large.
2016-06-01 05:50:21 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_lextents_j_curlextents,
Fix --disable-stats support. Fix numerous regressions that were exposed by --disable-stats, both in the core library and in the tests.
2017-01-19 15:03:37 +08:00
arenas_i(mib[2])->astats->lstats[mib[4]].curlextents, size_t)
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
static const ctl_named_node_t *
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
stats_arenas_i_lextents_j_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t j) {
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
if (j > SC_NSIZES - SC_NBINS) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return super_stats_arenas_i_lextents_j_node;
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
}
Add extents information to mallocstats output - Show number/bytes of extents of each size that are dirty, muzzy, retained.
2018-08-01 00:49:49 +08:00
CTL_RO_CGEN(config_stats, stats_arenas_i_extents_j_ndirty,
atomic_load_zu(
&arenas_i(mib[2])->astats->estats[mib[4]].ndirty,
ATOMIC_RELAXED), size_t);
CTL_RO_CGEN(config_stats, stats_arenas_i_extents_j_nmuzzy,
atomic_load_zu(
&arenas_i(mib[2])->astats->estats[mib[4]].nmuzzy,
ATOMIC_RELAXED), size_t);
CTL_RO_CGEN(config_stats, stats_arenas_i_extents_j_nretained,
atomic_load_zu(
&arenas_i(mib[2])->astats->estats[mib[4]].nretained,
ATOMIC_RELAXED), size_t);
CTL_RO_CGEN(config_stats, stats_arenas_i_extents_j_dirty_bytes,
atomic_load_zu(
&arenas_i(mib[2])->astats->estats[mib[4]].dirty_bytes,
ATOMIC_RELAXED), size_t);
CTL_RO_CGEN(config_stats, stats_arenas_i_extents_j_muzzy_bytes,
atomic_load_zu(
&arenas_i(mib[2])->astats->estats[mib[4]].muzzy_bytes,
ATOMIC_RELAXED), size_t);
CTL_RO_CGEN(config_stats, stats_arenas_i_extents_j_retained_bytes,
atomic_load_zu(
&arenas_i(mib[2])->astats->estats[mib[4]].retained_bytes,
ATOMIC_RELAXED), size_t);
static const ctl_named_node_t *
stats_arenas_i_extents_j_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t j) {
if (j >= SC_NPSIZES) {
return NULL;
}
return super_stats_arenas_i_extents_j_node;
}
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
static bool
ctl_arenas_i_verify(size_t i) {
size_t a = arenas_i2a_impl(i, true, true);
if (a == UINT_MAX || !ctl_arenas->arenas[a]->initialized) {
return true;
}
return false;
}
Add arena-specific and selective dss allocation. Add the "arenas.extend" mallctl, so that it is possible to create new arenas that are outside the set that jemalloc automatically multiplexes threads onto. Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible to explicitly allocate from a particular arena. Add the "opt.dss" mallctl, which controls the default precedence of dss allocation relative to mmap allocation. Add the "arena.<i>.dss" mallctl, which makes it possible to set the default dss precedence on a per arena or global basis. Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge". Add the "stats.arenas.<i>.dss" mallctl.
2012-10-12 04:53:15 +08:00
static const ctl_named_node_t *
Clean compilation -Wextra Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
2018-05-03 17:40:53 +08:00
stats_arenas_i_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t i) {
Add MALLCTL_ARENAS_ALL. Add the MALLCTL_ARENAS_ALL cpp macro as a fixed index for use in accessing the arena.<i>.{purge,decay,dss} and stats.arenas.<i>.* mallctls, and deprecate access via the arenas.narenas index (to be removed in 6.0.0).
2017-01-03 23:27:42 +08:00
const ctl_named_node_t *ret;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_lock(tsdn, &ctl_mtx);
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
if (ctl_arenas_i_verify(i)) {
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
ret = NULL;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
goto label_return;
Push down ctl_mtx. Many mallctl*() end points require no locking, so push the locking down to just the functions that need it. This is of particular import for "thread.allocated" and "thread.deallocated", which are intended as a low-overhead way to introspect per thread allocation activity.
2010-11-25 07:44:21 +08:00
}
ret = super_stats_arenas_i_node;
Rename labels. Rename labels from FOO to label_foo in order to avoid system macro definitions, in particular OUT and ERROR on mingw. Reported by Mike Hommey.
2012-04-11 06:07:44 +08:00
label_return:
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
malloc_mutex_unlock(tsdn, &ctl_mtx);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return ret;
Implement mallctl{nametomib,bymib}(). Replace chunk stats code that was missing locking; this fixes a race condition that could corrupt chunk statistics. Converting malloc_stats_print() to use mallctl*(). Add a missing semicolon in th DSS code. Convert malloc_tcache_flush() to a mallctl. Convert malloc_swap_enable() to a set of mallctl's.
2010-01-28 05:10:55 +08:00
}
Mallctl: Add experimental.hooks.[install|remove].
2018-04-25 05:45:41 +08:00
static int
experimental_hooks_install_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
if (oldp == NULL || oldlenp == NULL|| newp == NULL) {
ret = EINVAL;
goto label_return;
}
/*
* Note: this is a *private* struct. This is an experimental interface;
* forcing the user to know the jemalloc internals well enough to
* extract the ABI hopefully ensures nobody gets too comfortable with
* this API, which can change at a moment's notice.
*/
hooks_t hooks;
WRITE(hooks, hooks_t);
void *handle = hook_install(tsd_tsdn(tsd), &hooks);
if (handle == NULL) {
ret = EAGAIN;
goto label_return;
}
READ(handle, void *);
ret = 0;
label_return:
return ret;
}
static int
experimental_hooks_remove_ctl(tsd_t *tsd, const size_t *mib, size_t miblen,
void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
WRITEONLY();
void *handle = NULL;
WRITE(handle, void *);
if (handle == NULL) {
ret = EINVAL;
goto label_return;
}
hook_remove(tsd_tsdn(tsd), handle);
ret = 0;
label_return:
return ret;
}
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
/*
* Output six memory utilization entries for an input pointer, the first one of
* type (void *) and the remaining five of type size_t, describing the following
* (in the same order):
*
* (a) memory address of the extent a potential reallocation would go into,
* == the five fields below describe about the extent the pointer resides in ==
* (b) number of free regions in the extent,
* (c) number of regions in the extent,
* (d) size of the extent in terms of bytes,
* (e) total number of free regions in the bin the extent belongs to, and
* (f) total number of regions in the bin the extent belongs to.
*
* Note that "(e)" and "(f)" are only available when stats are enabled;
Improve memory utilization tests Added tests for large size classes and expanded the tests to cover wider range of allocation sizes.
2019-05-14 05:59:33 +08:00
* otherwise their values are undefined.
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
*
* This API is mainly intended for small class allocations, where extents are
Guard slabcur fetching in extent_util
2019-10-29 00:24:42 +08:00
* used as slab. Note that if the bin the extent belongs to is completely
* full, "(a)" will be NULL.
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
*
* In case of large class allocations, "(a)" will be NULL, and "(e)" and "(f)"
Improve memory utilization tests Added tests for large size classes and expanded the tests to cover wider range of allocation sizes.
2019-05-14 05:59:33 +08:00
* will be zero (if stats are enabled; otherwise undefined). The other three
* fields will be properly set though the values are trivial: "(b)" will be 0,
* "(c)" will be 1, and "(d)" will be the usable size.
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
*
* The input pointer and size are respectively passed in by newp and newlen,
* and the output fields and size are respectively oldp and *oldlenp.
*
* It can be beneficial to define the following macros to make it easier to
* access the output:
*
* #define SLABCUR_READ(out) (*(void **)out)
* #define COUNTS(out) ((size_t *)((void **)out + 1))
* #define NFREE_READ(out) COUNTS(out)[0]
* #define NREGS_READ(out) COUNTS(out)[1]
* #define SIZE_READ(out) COUNTS(out)[2]
* #define BIN_NFREE_READ(out) COUNTS(out)[3]
* #define BIN_NREGS_READ(out) COUNTS(out)[4]
*
* and then write e.g. NFREE_READ(oldp) to fetch the output. See the unit test
Separate tests for extent utilization API As title.
2019-04-10 02:01:26 +08:00
* test_query in test/unit/extent_util.c for an example.
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
*
* For a typical defragmentation workflow making use of this API for
* understanding the fragmentation level, please refer to the comment for
* experimental_utilization_batch_query_ctl.
*
* It's up to the application how to determine the significance of
* fragmentation relying on the outputs returned. Possible choices are:
*
* (a) if extent utilization ratio is below certain threshold,
* (b) if extent memory consumption is above certain threshold,
* (c) if extent utilization ratio is significantly below bin utilization ratio,
* (d) if input pointer deviates a lot from potential reallocation address, or
* (e) some selection/combination of the above.
*
* The caller needs to make sure that the input/output arguments are valid,
* in particular, that the size of the output is correct, i.e.:
*
* *oldlenp = sizeof(void *) + sizeof(size_t) * 5
*
* Otherwise, the function immediately returns EINVAL without touching anything.
*
* In the rare case where there's no associated extent found for the input
* pointer, the function zeros out all output fields and return. Please refer
* to the comment for experimental_utilization_batch_query_ctl to understand the
* motivation from C++.
*/
static int
experimental_utilization_query_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
assert(sizeof(inspect_extent_util_stats_verbose_t)
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
== sizeof(void *) + sizeof(size_t) * 5);
if (oldp == NULL || oldlenp == NULL
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
|| *oldlenp != sizeof(inspect_extent_util_stats_verbose_t)
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
|| newp == NULL) {
ret = EINVAL;
goto label_return;
}
void *ptr = NULL;
WRITE(ptr, void *);
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
inspect_extent_util_stats_verbose_t *util_stats
= (inspect_extent_util_stats_verbose_t *)oldp;
inspect_extent_util_stats_verbose_get(tsd_tsdn(tsd), ptr,
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
&util_stats->nfree, &util_stats->nregs, &util_stats->size,
&util_stats->bin_nfree, &util_stats->bin_nregs,
&util_stats->slabcur_addr);
ret = 0;
label_return:
return ret;
}
/*
* Given an input array of pointers, output three memory utilization entries of
* type size_t for each input pointer about the extent it resides in:
*
* (a) number of free regions in the extent,
* (b) number of regions in the extent, and
* (c) size of the extent in terms of bytes.
*
* This API is mainly intended for small class allocations, where extents are
* used as slab. In case of large class allocations, the outputs are trivial:
* "(a)" will be 0, "(b)" will be 1, and "(c)" will be the usable size.
*
* Note that multiple input pointers may reside on a same extent so the output
* fields may contain duplicates.
*
* The format of the input/output looks like:
*
* input[0]: 1st_pointer_to_query | output[0]: 1st_extent_n_free_regions
* | output[1]: 1st_extent_n_regions
* | output[2]: 1st_extent_size
* input[1]: 2nd_pointer_to_query | output[3]: 2nd_extent_n_free_regions
* | output[4]: 2nd_extent_n_regions
* | output[5]: 2nd_extent_size
* ... | ...
*
* The input array and size are respectively passed in by newp and newlen, and
* the output array and size are respectively oldp and *oldlenp.
*
* It can be beneficial to define the following macros to make it easier to
* access the output:
*
* #define NFREE_READ(out, i) out[(i) * 3]
* #define NREGS_READ(out, i) out[(i) * 3 + 1]
* #define SIZE_READ(out, i) out[(i) * 3 + 2]
*
* and then write e.g. NFREE_READ(oldp, i) to fetch the output. See the unit
Separate tests for extent utilization API As title.
2019-04-10 02:01:26 +08:00
* test test_batch in test/unit/extent_util.c for a concrete example.
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
*
* A typical workflow would be composed of the following steps:
*
* (1) flush tcache: mallctl("thread.tcache.flush", ...)
* (2) initialize input array of pointers to query fragmentation
* (3) allocate output array to hold utilization statistics
* (4) query utilization: mallctl("experimental.utilization.batch_query", ...)
* (5) (optional) decide if it's worthwhile to defragment; otherwise stop here
* (6) disable tcache: mallctl("thread.tcache.enabled", ...)
* (7) defragment allocations with significant fragmentation, e.g.:
* for each allocation {
* if it's fragmented {
* malloc(...);
* memcpy(...);
* free(...);
* }
* }
* (8) enable tcache: mallctl("thread.tcache.enabled", ...)
*
* The application can determine the significance of fragmentation themselves
* relying on the statistics returned, both at the overall level i.e. step "(5)"
* and at individual allocation level i.e. within step "(7)". Possible choices
* are:
*
* (a) whether memory utilization ratio is below certain threshold,
* (b) whether memory consumption is above certain threshold, or
* (c) some combination of the two.
*
* The caller needs to make sure that the input/output arrays are valid and
* their sizes are proper as well as matched, meaning:
*
* (a) newlen = n_pointers * sizeof(const void *)
* (b) *oldlenp = n_pointers * sizeof(size_t) * 3
* (c) n_pointers > 0
*
* Otherwise, the function immediately returns EINVAL without touching anything.
*
* In the rare case where there's no associated extent found for some pointers,
* rather than immediately terminating the computation and raising an error,
* the function simply zeros out the corresponding output fields and continues
* the computation until all input pointers are handled. The motivations of
* such a design are as follows:
*
* (a) The function always either processes nothing or processes everything, and
* never leaves the output half touched and half untouched.
*
* (b) It facilitates usage needs especially common in C++. A vast variety of
* C++ objects are instantiated with multiple dynamic memory allocations. For
* example, std::string and std::vector typically use at least two allocations,
* one for the metadata and one for the actual content. Other types may use
* even more allocations. When inquiring about utilization statistics, the
* caller often wants to examine into all such allocations, especially internal
* one(s), rather than just the topmost one. The issue comes when some
* implementations do certain optimizations to reduce/aggregate some internal
* allocations, e.g. putting short strings directly into the metadata, and such
* decisions are not known to the caller. Therefore, we permit pointers to
* memory usages that may not be returned by previous malloc calls, and we
* provide the caller a convenient way to identify such cases.
*/
static int
experimental_utilization_batch_query_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
assert(sizeof(inspect_extent_util_stats_t) == sizeof(size_t) * 3);
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
const size_t len = newlen / sizeof(const void *);
if (oldp == NULL || oldlenp == NULL || newp == NULL || newlen == 0
|| newlen != len * sizeof(const void *)
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
|| *oldlenp != len * sizeof(inspect_extent_util_stats_t)) {
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
ret = EINVAL;
goto label_return;
}
void **ptrs = (void **)newp;
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
inspect_extent_util_stats_t *util_stats =
(inspect_extent_util_stats_t *)oldp;
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
size_t i;
for (i = 0; i < len; ++i) {
Extents: Split out introspection functionality. This isn't really part of the core extent allocation facilities. Especially as this module grows, having it in its own place may come in handy.
2019-12-05 01:44:59 +08:00
inspect_extent_util_stats_get(tsd_tsdn(tsd), ptrs[i],
Add memory utilization analytics to mallctl The analytics tool is put under experimental.utilization namespace in mallctl. Input is one pointer or an array of pointers and the output is a list of memory utilization statistics.
2019-03-16 02:01:45 +08:00
&util_stats[i].nfree, &util_stats[i].nregs,
&util_stats[i].size);
}
ret = 0;
label_return:
return ret;
}
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
static const ctl_named_node_t *
experimental_arenas_i_index(tsdn_t *tsdn, const size_t *mib,
size_t miblen, size_t i) {
const ctl_named_node_t *ret;
malloc_mutex_lock(tsdn, &ctl_mtx);
if (ctl_arenas_i_verify(i)) {
ret = NULL;
goto label_return;
}
ret = super_experimental_arenas_i_node;
label_return:
malloc_mutex_unlock(tsdn, &ctl_mtx);
return ret;
}
static int
experimental_arenas_i_pactivep_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
if (!config_stats) {
return ENOENT;
}
if (oldp == NULL || oldlenp == NULL || *oldlenp != sizeof(size_t *)) {
return EINVAL;
}
unsigned arena_ind;
arena_t *arena;
int ret;
size_t *pactivep;
malloc_mutex_lock(tsd_tsdn(tsd), &ctl_mtx);
READONLY();
MIB_UNSIGNED(arena_ind, 2);
if (arena_ind < narenas_total_get() && (arena =
arena_get(tsd_tsdn(tsd), arena_ind, false)) != NULL) {
#if defined(JEMALLOC_GCC_ATOMIC_ATOMICS) || \
defined(JEMALLOC_GCC_SYNC_ATOMICS) || defined(_MSC_VER)
/* Expose the underlying counter for fast read. */
PA: Move in nactive counter.
2020-03-12 08:40:17 +08:00
pactivep = (size_t *)&(arena->pa_shard.nactive.repr);
Add experimental.arenas.i.pactivep. The new experimental mallctl exposes the arena pactive counter to applications, which allows fast read w/o going through the mallctl / epoch steps. This is particularly useful when frequent balancing is required, e.g. when having multiple manual arenas, and threads are multiplexed to them based on usage.
2019-05-21 04:41:43 +08:00
READ(pactivep, size_t *);
ret = 0;
#else
ret = EFAULT;
#endif
} else {
ret = EFAULT;
}
label_return:
malloc_mutex_unlock(tsd_tsdn(tsd), &ctl_mtx);
return ret;
}
Last-N profiling mode
2019-12-19 05:38:14 +08:00
static int
experimental_prof_recent_alloc_max_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
if (!(config_prof && opt_prof)) {
ret = ENOENT;
goto label_return;
}
ssize_t old_max;
if (newp != NULL) {
if (newlen != sizeof(ssize_t)) {
ret = EINVAL;
goto label_return;
}
ssize_t max = *(ssize_t *)newp;
if (max < -1) {
ret = EINVAL;
goto label_return;
}
old_max = prof_recent_alloc_max_ctl_write(tsd, max);
} else {
old_max = prof_recent_alloc_max_ctl_read();
}
READ(old_max, ssize_t);
ret = 0;
label_return:
return ret;
}
Add mallctl for dumping last-N profiling records
2020-02-05 08:05:11 +08:00
typedef struct write_cb_packet_s write_cb_packet_t;
struct write_cb_packet_s {
void (*write_cb)(void *, const char *);
void *cbopaque;
};
static int
experimental_prof_recent_alloc_dump_ctl(tsd_t *tsd, const size_t *mib,
size_t miblen, void *oldp, size_t *oldlenp, void *newp, size_t newlen) {
int ret;
if (!(config_prof && opt_prof)) {
ret = ENOENT;
goto label_return;
}
assert(sizeof(write_cb_packet_t) == sizeof(void *) * 2);
WRITEONLY();
write_cb_packet_t write_cb_packet;
ASSURED_WRITE(write_cb_packet, write_cb_packet_t);
prof_recent_alloc_dump(tsd, write_cb_packet.write_cb,
write_cb_packet.cbopaque);
ret = 0;
label_return:
return ret;
}
Reference in New Issue Copy Permalink