project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
36becb1302
server-skynet-source-3rd-je.../include/jemalloc/internal/arena_inlines_b.h

608 lines
18 KiB
C
Raw Normal View History

Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#ifndef JEMALLOC_INTERNAL_ARENA_INLINES_B_H
#define JEMALLOC_INTERNAL_ARENA_INLINES_B_H
Make headers self-contained by fixing `#include`s Header files are now self-contained, which makes the relationships between the files clearer, and crucially allows LSP tools like `clangd` to function correctly in all of our header files. I have verified that the headers are self-contained (aside from the various Windows shims) by compiling them as if they were C files – in a follow-up commit I plan to add this to CI to ensure we don't regress on this front.
2023-06-10 08:37:47 +08:00
#include "jemalloc/internal/jemalloc_preamble.h"
#include "jemalloc/internal/arena_externs.h"
#include "jemalloc/internal/arena_structs.h"
Tcache flush: keep common path state in registers. By carefully force-inlining the division constants and the operation sum count, we can eliminate redundant operations in the arena-level dalloc function. Do so.
2021-01-30 08:06:28 +08:00
#include "jemalloc/internal/div.h"
Emap: Move in iealloc. This is logically scoped to the emap.
2020-02-06 06:50:34 +08:00
#include "jemalloc/internal/emap.h"
Make headers self-contained by fixing `#include`s Header files are now self-contained, which makes the relationships between the files clearer, and crucially allows LSP tools like `clangd` to function correctly in all of our header files. I have verified that the headers are self-contained (aside from the various Windows shims) by compiling them as if they were C files – in a follow-up commit I plan to add this to CI to ensure we don't regress on this front.
2023-06-10 08:37:47 +08:00
#include "jemalloc/internal/jemalloc_internal_inlines_b.h"
Header refactoring: move jemalloc_internal_types.h out of the catch-all
2017-04-18 06:52:44 +08:00
#include "jemalloc/internal/jemalloc_internal_types.h"
Make headers self-contained by fixing `#include`s Header files are now self-contained, which makes the relationships between the files clearer, and crucially allows LSP tools like `clangd` to function correctly in all of our header files. I have verified that the headers are self-contained (aside from the various Windows shims) by compiling them as if they were C files – in a follow-up commit I plan to add this to CI to ensure we don't regress on this front.
2023-06-10 08:37:47 +08:00
#include "jemalloc/internal/large_externs.h"
Header refactoring: unify and de-catchall mutex module
2017-05-24 03:28:19 +08:00
#include "jemalloc/internal/mutex.h"
Make headers self-contained by fixing `#include`s Header files are now self-contained, which makes the relationships between the files clearer, and crucially allows LSP tools like `clangd` to function correctly in all of our header files. I have verified that the headers are self-contained (aside from the various Windows shims) by compiling them as if they were C files – in a follow-up commit I plan to add this to CI to ensure we don't regress on this front.
2023-06-10 08:37:47 +08:00
#include "jemalloc/internal/prof_externs.h"
#include "jemalloc/internal/prof_structs.h"
Header refactoring: unify and de-catchall rtree module.
2017-05-24 05:26:31 +08:00
#include "jemalloc/internal/rtree.h"
Eagerly detect double free and sized dealloc bugs for large sizes.
2020-10-15 07:45:19 +08:00
#include "jemalloc/internal/safety_check.h"
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
#include "jemalloc/internal/sc.h"
Header refactoring: Pull size helpers out of jemalloc module.
2017-05-31 01:45:37 +08:00
#include "jemalloc/internal/sz.h"
Make headers self-contained by fixing `#include`s Header files are now self-contained, which makes the relationships between the files clearer, and crucially allows LSP tools like `clangd` to function correctly in all of our header files. I have verified that the headers are self-contained (aside from the various Windows shims) by compiling them as if they were C files – in a follow-up commit I plan to add this to CI to ensure we don't regress on this front.
2023-06-10 08:37:47 +08:00
#include "jemalloc/internal/tcache_inlines.h"
Header refactoring: ticker module - remove from the catchall and unify.
2017-04-20 04:39:33 +08:00
#include "jemalloc/internal/ticker.h"
Header refactoring: move jemalloc_internal_types.h out of the catch-all
2017-04-18 06:52:44 +08:00
Arena: Add helper function arena_get_from_extent.
2019-09-24 09:05:57 +08:00
static inline arena_t *
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
arena_get_from_edata(edata_t *edata) {
return (arena_t *)atomic_load_p(&arenas[edata_arena_ind_get(edata)],
Arena: Add helper function arena_get_from_extent.
2019-09-24 09:05:57 +08:00
ATOMIC_RELAXED);
}
Implement huge arena: opt.huge_threshold. The feature allows using a dedicated arena for huge allocations. We want the addtional arena to separate huge allocation because: 1) mixing small extents with huge ones causes fragmentation over the long run (this feature reduces VM size significantly); 2) with many arenas, huge extents rarely get reused across threads; and 3) huge allocations happen way less frequently, therefore no concerns for lock contention.
2018-05-22 04:33:48 +08:00
JEMALLOC_ALWAYS_INLINE arena_t *
arena_choose_maybe_huge(tsd_t *tsd, arena_t *arena, size_t size) {
if (arena != NULL) {
return arena;
}
/*
* For huge allocations, use the dedicated huge arena if both are true:
* 1) is using auto arena selection (i.e. arena == NULL), and 2) the
* thread is not assigned to a manual arena.
*/
Fix the arena selection for oversized allocations. Use the per-arena oversize_threshold, instead of the global setting.
2023-06-07 04:37:07 +08:00
arena_t *tsd_arena = tsd_arena_get(tsd);
if (tsd_arena == NULL) {
tsd_arena = arena_choose(tsd, NULL);
Implement huge arena: opt.huge_threshold. The feature allows using a dedicated arena for huge allocations. We want the addtional arena to separate huge allocation because: 1) mixing small extents with huge ones causes fragmentation over the long run (this feature reduces VM size significantly); 2) with many arenas, huge extents rarely get reused across threads; and 3) huge allocations happen way less frequently, therefore no concerns for lock contention.
2018-05-22 04:33:48 +08:00
}
Fix the arena selection for oversized allocations. Use the per-arena oversize_threshold, instead of the global setting.
2023-06-07 04:37:07 +08:00
size_t threshold = atomic_load_zu(
&tsd_arena->pa_shard.pac.oversize_threshold, ATOMIC_RELAXED);
if (unlikely(size >= threshold) && arena_is_auto(tsd_arena)) {
return arena_choose_huge(tsd);
}
return tsd_arena;
Implement huge arena: opt.huge_threshold. The feature allows using a dedicated arena for huge allocations. We want the addtional arena to separate huge allocation because: 1) mixing small extents with huge ones causes fragmentation over the long run (this feature reduces VM size significantly); 2) with many arenas, huge extents rarely get reused across threads; and 3) huge allocations happen way less frequently, therefore no concerns for lock contention.
2018-05-22 04:33:48 +08:00
}
Fix segfault in rb `*_tree_remove` Static analysis flagged this. It's possible to segfault in the `*_tree_remove` function generated by `rb_gen`, as `nodep` may still be `NULL` after the initial for loop. I can confirm from reviewing the fleetwide coredump data that this was in fact being hit in production, primarily through `tctx_tree_remove`, and much more rarely through `gctx_tree_remove`.
2023-05-10 03:06:47 +08:00
JEMALLOC_ALWAYS_INLINE bool
large_dalloc_safety_checks(edata_t *edata, const void *ptr, szind_t szind) {
if (!config_opt_safety_checks) {
return false;
}
/*
* Eagerly detect double free and sized dealloc bugs for large sizes.
* The cost is low enough (as edata will be accessed anyway) to be
* enabled all the time.
*/
if (unlikely(edata == NULL ||
edata_state_get(edata) != extent_state_active)) {
safety_check_fail("Invalid deallocation detected: "
"pages being freed (%p) not currently active, "
"possibly caused by double free bugs.", ptr);
return true;
}
size_t input_size = sz_index2size(szind);
if (unlikely(input_size != edata_usize_get(edata))) {
safety_check_fail_sized_dealloc(/* current_dealloc */ true, ptr,
/* true_size */ edata_usize_get(edata), input_size);
return true;
}
return false;
}
Restructure profiling Develop new data structure and code logic for holding profiling related information stored in the extent that may be needed after the extent is released, which in particular is the case for the reallocation code path (e.g. in `rallocx()` and `xallocx()`). The data structure is a generalization of `prof_tctx_t`: we previously only copy out the `prof_tctx` before the extent is released, but we may be in need of additional fields. Currently the only additional field is the allocation time field, but there may be more fields in the future. The restructuring also resolved a bug: `prof_realloc()` mistakenly passed the new `ptr` to `prof_free_sampled_object()`, but passing in the `old_ptr` would crash because it's already been released. Now the essential profiling information is collectively copied out early and safely passed to `prof_free_sampled_object()` after the extent is released.
2019-11-20 08:24:57 +08:00
JEMALLOC_ALWAYS_INLINE void
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
arena_prof_info_get(tsd_t *tsd, const void *ptr, emap_alloc_ctx_t *alloc_ctx,
Last-N profiling mode
2019-12-19 05:38:14 +08:00
prof_info_t *prof_info, bool reset_recent) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
cassert(config_prof);
assert(ptr != NULL);
Restructure profiling Develop new data structure and code logic for holding profiling related information stored in the extent that may be needed after the extent is released, which in particular is the case for the reallocation code path (e.g. in `rallocx()` and `xallocx()`). The data structure is a generalization of `prof_tctx_t`: we previously only copy out the `prof_tctx` before the extent is released, but we may be in need of additional fields. Currently the only additional field is the allocation time field, but there may be more fields in the future. The restructuring also resolved a bug: `prof_realloc()` mistakenly passed the new `ptr` to `prof_free_sampled_object()`, but passing in the `old_ptr` would crash because it's already been released. Now the essential profiling information is collectively copied out early and safely passed to `prof_free_sampled_object()` after the extent is released.
2019-11-20 08:24:57 +08:00
assert(prof_info != NULL);
Last-N profiling mode
2019-12-19 05:38:14 +08:00
edata_t *edata = NULL;
Restructure profiling Develop new data structure and code logic for holding profiling related information stored in the extent that may be needed after the extent is released, which in particular is the case for the reallocation code path (e.g. in `rallocx()` and `xallocx()`). The data structure is a generalization of `prof_tctx_t`: we previously only copy out the `prof_tctx` before the extent is released, but we may be in need of additional fields. Currently the only additional field is the allocation time field, but there may be more fields in the future. The restructuring also resolved a bug: `prof_realloc()` mistakenly passed the new `ptr` to `prof_free_sampled_object()`, but passing in the `old_ptr` would crash because it's already been released. Now the essential profiling information is collectively copied out early and safely passed to `prof_free_sampled_object()` after the extent is released.
2019-11-20 08:24:57 +08:00
bool is_slab;
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Pass alloc_ctx down profiling path. With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
2017-04-12 09:13:10 +08:00
/* Static check. */
if (alloc_ctx == NULL) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata = emap_edata_lookup(tsd_tsdn(tsd), &arena_emap_global,
ptr);
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
is_slab = edata_slab_get(edata);
Fix unlikely condition in arena_prof_info_get()
2019-12-25 03:30:23 +08:00
} else if (unlikely(!(is_slab = alloc_ctx->slab))) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata = emap_edata_lookup(tsd_tsdn(tsd), &arena_emap_global,
ptr);
Restructure profiling Develop new data structure and code logic for holding profiling related information stored in the extent that may be needed after the extent is released, which in particular is the case for the reallocation code path (e.g. in `rallocx()` and `xallocx()`). The data structure is a generalization of `prof_tctx_t`: we previously only copy out the `prof_tctx` before the extent is released, but we may be in need of additional fields. Currently the only additional field is the allocation time field, but there may be more fields in the future. The restructuring also resolved a bug: `prof_realloc()` mistakenly passed the new `ptr` to `prof_free_sampled_object()`, but passing in the `old_ptr` would crash because it's already been released. Now the essential profiling information is collectively copied out early and safely passed to `prof_free_sampled_object()` after the extent is released.
2019-11-20 08:24:57 +08:00
}
if (unlikely(!is_slab)) {
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
/* edata must have been initialized at this point. */
Last-N profiling mode
2019-12-19 05:38:14 +08:00
assert(edata != NULL);
Fix segfault in rb `*_tree_remove` Static analysis flagged this. It's possible to segfault in the `*_tree_remove` function generated by `rb_gen`, as `nodep` may still be `NULL` after the initial for loop. I can confirm from reviewing the fleetwide coredump data that this was in fact being hit in production, primarily through `tctx_tree_remove`, and much more rarely through `gctx_tree_remove`.
2023-05-10 03:06:47 +08:00
if (reset_recent &&
large_dalloc_safety_checks(edata, ptr,
edata_szind_get(edata))) {
Define `PROF_TCTX_SENTINEL` instead of using magic numbers This makes the code more readable on its own, and also sets the stage for more cleanly handling the pointer provenance lints in a following commit.
2023-07-22 09:13:58 +08:00
prof_info->alloc_tctx = PROF_TCTX_SENTINEL;
Fix segfault in rb `*_tree_remove` Static analysis flagged this. It's possible to segfault in the `*_tree_remove` function generated by `rb_gen`, as `nodep` may still be `NULL` after the initial for loop. I can confirm from reviewing the fleetwide coredump data that this was in fact being hit in production, primarily through `tctx_tree_remove`, and much more rarely through `gctx_tree_remove`.
2023-05-10 03:06:47 +08:00
return;
}
Last-N profiling mode
2019-12-19 05:38:14 +08:00
large_prof_info_get(tsd, edata, prof_info, reset_recent);
Pass alloc_ctx down profiling path. With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
2017-04-12 09:13:10 +08:00
} else {
Define `PROF_TCTX_SENTINEL` instead of using magic numbers This makes the code more readable on its own, and also sets the stage for more cleanly handling the pointer provenance lints in a following commit.
2023-07-22 09:13:58 +08:00
prof_info->alloc_tctx = PROF_TCTX_SENTINEL;
Last-N profiling mode
2019-12-19 05:38:14 +08:00
/*
* No need to set other fields in prof_info; they will never be
Define `PROF_TCTX_SENTINEL` instead of using magic numbers This makes the code more readable on its own, and also sets the stage for more cleanly handling the pointer provenance lints in a following commit.
2023-07-22 09:13:58 +08:00
* accessed if alloc_tctx == PROF_TCTX_SENTINEL.
Last-N profiling mode
2019-12-19 05:38:14 +08:00
*/
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
Pass alloc_ctx down profiling path. With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
2017-04-12 09:13:10 +08:00
JEMALLOC_ALWAYS_INLINE void
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
arena_prof_tctx_reset(tsd_t *tsd, const void *ptr,
emap_alloc_ctx_t *alloc_ctx) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
cassert(config_prof);
assert(ptr != NULL);
Pass alloc_ctx down profiling path. With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
2017-04-12 09:13:10 +08:00
/* Static check. */
if (alloc_ctx == NULL) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsd_tsdn(tsd),
&arena_emap_global, ptr);
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
if (unlikely(!edata_slab_get(edata))) {
large_prof_tctx_reset(edata);
Pass alloc_ctx down profiling path. With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
2017-04-12 09:13:10 +08:00
}
} else {
if (unlikely(!alloc_ctx->slab)) {
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
edata_t *edata = emap_edata_lookup(tsd_tsdn(tsd),
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
&arena_emap_global, ptr);
Emap: Move in iealloc. This is logically scoped to the emap.
2020-02-06 06:50:34 +08:00
large_prof_tctx_reset(edata);
Pass alloc_ctx down profiling path. With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
2017-04-12 09:13:10 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
Restructure setters for profiling info Explicitly define three setters: - `prof_tctx_reset()`: set `prof_tctx` to `1U`, if we don't know in advance whether the allocation is large or not; - `prof_tctx_reset_sampled()`: set `prof_tctx` to `1U`, if we already know in advance that the allocation is large; - `prof_info_set()`: set a real `prof_tctx`, and also set other profiling info e.g. the allocation time. Code structure wise, the prof level is kept as a thin wrapper, the large level only provides low level setter APIs, and the arena level carries out the main logic.
2019-12-14 08:48:03 +08:00
JEMALLOC_ALWAYS_INLINE void
arena_prof_tctx_reset_sampled(tsd_t *tsd, const void *ptr) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
cassert(config_prof);
assert(ptr != NULL);
Push down iealloc() calls. Call iealloc() as deep into call chains as possible without causing redundant calls.
2017-03-21 02:00:07 +08:00
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsd_tsdn(tsd), &arena_emap_global,
ptr);
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
assert(!edata_slab_get(edata));
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
large_prof_tctx_reset(edata);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
JEMALLOC_ALWAYS_INLINE void
Record request size in prof info
2020-08-22 02:31:53 +08:00
arena_prof_info_set(tsd_t *tsd, edata_t *edata, prof_tctx_t *tctx,
size_t size) {
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
cassert(config_prof);
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
assert(!edata_slab_get(edata));
Record request size in prof info
2020-08-22 02:31:53 +08:00
large_prof_info_set(edata, tctx, size);
Add logging for sampled allocations - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
2018-07-06 01:56:33 +08:00
}
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
JEMALLOC_ALWAYS_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arena_decay_ticks(tsdn_t *tsdn, arena_t *arena, unsigned nticks) {
if (unlikely(tsdn_null(tsdn))) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Use ticker_geom_t for arena tcache decay.
2021-02-01 04:50:55 +08:00
tsd_t *tsd = tsdn_tsd(tsdn);
/*
* We use the ticker_geom_t to avoid having per-arena state in the tsd.
* Instead of having a countdown-until-decay timer running for every
* arena in every thread, we flip a coin once per tick, whose
* probability of coming up heads is 1/nticks; this is effectively the
* operation of the ticker_geom_t. Each arena has the same chance of a
* coinflip coming up heads (1/ARENA_DECAY_NTICKS_PER_UPDATE), so we can
* use a single ticker for all of them.
*/
ticker_geom_t *decay_ticker = tsd_arena_decay_tickerp_get(tsd);
uint64_t *prng_state = tsd_prng_statep_get(tsd);
Disallow decay during reentrancy. Decay should not be triggered during reentrant calls (may cause lock order reversal / deadlocks). Added a delay_trigger flag to the tickers to bypass decay when rentrancy_level is not zero.
2023-03-31 10:02:24 +08:00
if (unlikely(ticker_geom_ticks(decay_ticker, prng_state, nticks,
tsd_reentrancy_level_get(tsd) > 0))) {
Implementing opt.background_thread. Added opt.background_thread to enable background threads, which handles purging currently. When enabled, decay ticks will not trigger purging (which will be left to the background threads). We limit the max number of threads to NCPUs. When percpu arena is enabled, set CPU affinity for the background threads as well. The sleep interval of background threads is dynamic and determined by computing number of pages to purge in the future (based on backlog).
2017-03-18 03:42:33 +08:00
arena_decay(tsdn, arena, false, false);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
JEMALLOC_ALWAYS_INLINE void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arena_decay_tick(tsdn_t *tsdn, arena_t *arena) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
arena_decay_ticks(tsdn, arena, 1);
}
JEMALLOC_ALWAYS_INLINE void *
arena_malloc(tsdn_t *tsdn, arena_t *arena, size_t size, szind_t ind, bool zero,
Reduce the memory overhead of sampled small allocations Previously, small allocations which were sampled as part of heap profiling were rounded up to `SC_LARGE_MINCLASS`. This additional memory usage becomes problematic when the page size is increased, as noted in #2358. Small allocations are now rounded up to the nearest multiple of `PAGE` instead, reducing the memory overhead by a factor of 4 in the most extreme cases.
2023-06-03 06:15:37 +08:00
bool slab, tcache_t *tcache, bool slow_path) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
assert(!tsdn_null(tsdn) || tcache == NULL);
if (likely(tcache != NULL)) {
Reduce the memory overhead of sampled small allocations Previously, small allocations which were sampled as part of heap profiling were rounded up to `SC_LARGE_MINCLASS`. This additional memory usage becomes problematic when the page size is increased, as noted in #2358. Small allocations are now rounded up to the nearest multiple of `PAGE` instead, reducing the memory overhead by a factor of 4 in the most extreme cases.
2023-06-03 06:15:37 +08:00
if (likely(slab)) {
assert(sz_can_use_slab(size));
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return tcache_alloc_small(tsdn_tsd(tsdn), arena,
tcache, size, ind, zero, slow_path);
Enable per-tcache tcache_max 1. add tcache_max and nhbins into tcache_t so that they are per-tcache, with one auto tcache per thread, it's also per-thread; 2. add mallctl for each thread to set its own tcache_max (of its auto tcache); 3. store the maximum number of items in each bin instead of using a global storage; 4. add tests for the modifications above. 5. Rename `nhbins` and `tcache_maxclass` to `global_do_not_change_nhbins` and `global_do_not_change_tcache_maxclass`.
2023-08-07 02:38:30 +08:00
} else if (likely(size <= tcache_max_get(tcache))) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return tcache_alloc_large(tsdn_tsd(tsdn), arena,
tcache, size, ind, zero, slow_path);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
Enable per-tcache tcache_max 1. add tcache_max and nhbins into tcache_t so that they are per-tcache, with one auto tcache per thread, it's also per-thread; 2. add mallctl for each thread to set its own tcache_max (of its auto tcache); 3. store the maximum number of items in each bin instead of using a global storage; 4. add tests for the modifications above. 5. Rename `nhbins` and `tcache_maxclass` to `global_do_not_change_nhbins` and `global_do_not_change_tcache_maxclass`.
2023-08-07 02:38:30 +08:00
/* (size > tcache_max) case falls through. */
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
Reduce the memory overhead of sampled small allocations Previously, small allocations which were sampled as part of heap profiling were rounded up to `SC_LARGE_MINCLASS`. This additional memory usage becomes problematic when the page size is increased, as noted in #2358. Small allocations are now rounded up to the nearest multiple of `PAGE` instead, reducing the memory overhead by a factor of 4 in the most extreme cases.
2023-06-03 06:15:37 +08:00
return arena_malloc_hard(tsdn, arena, size, ind, zero, slab);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
JEMALLOC_ALWAYS_INLINE arena_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
arena_aalloc(tsdn_t *tsdn, const void *ptr) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global, ptr);
Emap: Move in iealloc. This is logically scoped to the emap.
2020-02-06 06:50:34 +08:00
unsigned arena_ind = edata_arena_ind_get(edata);
return (arena_t *)atomic_load_p(&arenas[arena_ind], ATOMIC_RELAXED);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
JEMALLOC_ALWAYS_INLINE size_t
Remove extent arg from isalloc() and arena_salloc().
2017-03-17 16:25:12 +08:00
arena_salloc(tsdn_t *tsdn, const void *ptr) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
assert(ptr != NULL);
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t alloc_ctx;
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
emap_alloc_ctx_lookup(tsdn, &arena_emap_global, ptr, &alloc_ctx);
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(alloc_ctx.szind != SC_NSIZES);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
return sz_index2size(alloc_ctx.szind);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
}
JEMALLOC_ALWAYS_INLINE size_t
arena_vsalloc(tsdn_t *tsdn, const void *ptr) {
/*
* Return 0 if ptr is not within an extent managed by jemalloc. This
* function has two extra costs relative to isalloc():
* - The rtree calls cannot claim to be dependent lookups, which induces
* rtree lookup load dependencies.
* - The lookup may fail, so there is an extra branch to check for
* failure.
*/
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
emap_full_alloc_ctx_t full_alloc_ctx;
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
bool missing = emap_full_alloc_ctx_try_lookup(tsdn, &arena_emap_global,
ptr, &full_alloc_ctx);
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (missing) {
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
return 0;
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (full_alloc_ctx.edata == NULL) {
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
return 0;
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(edata_state_get(full_alloc_ctx.edata) == extent_state_active);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
/* Only slab members should be looked up via interior pointers. */
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(edata_addr_get(full_alloc_ctx.edata) == ptr
|| edata_slab_get(full_alloc_ctx.edata));
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(full_alloc_ctx.szind != SC_NSIZES);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
return sz_index2size(full_alloc_ctx.szind);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
Invoke arena_dalloc_promoted() properly w/o tcache. When tcache was disabled, the dalloc promoted case was missing.
2019-07-25 07:12:06 +08:00
static inline void
arena_dalloc_large_no_tcache(tsdn_t *tsdn, void *ptr, szind_t szind) {
if (config_prof && unlikely(szind < SC_NBINS)) {
arena_dalloc_promoted(tsdn, ptr, NULL, true);
} else {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global,
ptr);
Report the offending pointer on sized dealloc bug detection.
2021-02-06 08:47:09 +08:00
if (large_dalloc_safety_checks(edata, ptr, szind)) {
Eagerly detect double free and sized dealloc bugs for large sizes.
2020-10-15 07:45:19 +08:00
/* See the comment in isfree. */
return;
}
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
large_dalloc(tsdn, edata);
Invoke arena_dalloc_promoted() properly w/o tcache. When tcache was disabled, the dalloc promoted case was missing.
2019-07-25 07:12:06 +08:00
}
}
Get rid of most of the various inline macros.
2017-04-22 00:37:34 +08:00
static inline void
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
arena_dalloc_no_tcache(tsdn_t *tsdn, void *ptr) {
assert(ptr != NULL);
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t alloc_ctx;
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
emap_alloc_ctx_lookup(tsdn, &arena_emap_global, ptr, &alloc_ctx);
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
if (config_debug) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global,
ptr);
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(alloc_ctx.szind == edata_szind_get(edata));
assert(alloc_ctx.szind < SC_NSIZES);
assert(alloc_ctx.slab == edata_slab_get(edata));
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (likely(alloc_ctx.slab)) {
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
/* Small allocation. */
arena_dalloc_small(tsdn, ptr);
} else {
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
arena_dalloc_large_no_tcache(tsdn, ptr, alloc_ctx.szind);
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
}
}
Refactor arena_dalloc() / _sdalloc().
2019-07-25 07:27:30 +08:00
JEMALLOC_ALWAYS_INLINE void
arena_dalloc_large(tsdn_t *tsdn, void *ptr, tcache_t *tcache, szind_t szind,
bool slow_path) {
Enable per-tcache tcache_max 1. add tcache_max and nhbins into tcache_t so that they are per-tcache, with one auto tcache per thread, it's also per-thread; 2. add mallctl for each thread to set its own tcache_max (of its auto tcache); 3. store the maximum number of items in each bin instead of using a global storage; 4. add tests for the modifications above. 5. Rename `nhbins` and `tcache_maxclass` to `global_do_not_change_nhbins` and `global_do_not_change_tcache_maxclass`.
2023-08-07 02:38:30 +08:00
assert (!tsdn_null(tsdn) && tcache != NULL);
if (szind < tcache_nhbins_get(tcache)) {
Refactor arena_dalloc() / _sdalloc().
2019-07-25 07:27:30 +08:00
if (config_prof && unlikely(szind < SC_NBINS)) {
arena_dalloc_promoted(tsdn, ptr, tcache, slow_path);
} else {
tcache_dalloc_large(tsdn_tsd(tsdn), tcache, ptr, szind,
slow_path);
}
} else {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global,
ptr);
Report the offending pointer on sized dealloc bug detection.
2021-02-06 08:47:09 +08:00
if (large_dalloc_safety_checks(edata, ptr, szind)) {
Eagerly detect double free and sized dealloc bugs for large sizes.
2020-10-15 07:45:19 +08:00
/* See the comment in isfree. */
return;
}
Rename extent_t to edata_t. This frees us up from the unfortunate extent/extent2 naming collision.
2019-12-10 06:36:45 +08:00
large_dalloc(tsdn, edata);
Refactor arena_dalloc() / _sdalloc().
2019-07-25 07:27:30 +08:00
}
}
Add double free detection using slab bitmap for debug build Add a sanity check for double free issue in the arena in case that the tcache has been flushed.
2022-08-10 07:39:02 +08:00
/* Find the region index of a pointer. */
JEMALLOC_ALWAYS_INLINE size_t
arena_slab_regind_impl(div_info_t* div_info, szind_t binind,
edata_t *slab, const void *ptr) {
size_t diff, regind;
/* Freeing a pointer outside the slab can cause assertion failure. */
assert((uintptr_t)ptr >= (uintptr_t)edata_addr_get(slab));
assert((uintptr_t)ptr < (uintptr_t)edata_past_get(slab));
/* Freeing an interior pointer can cause assertion failure. */
assert(((uintptr_t)ptr - (uintptr_t)edata_addr_get(slab)) %
(uintptr_t)bin_infos[binind].reg_size == 0);
diff = (size_t)((uintptr_t)ptr - (uintptr_t)edata_addr_get(slab));
/* Avoid doing division with a variable divisor. */
regind = div_compute(div_info, diff);
assert(regind < bin_infos[binind].nregs);
return regind;
}
/* Checks whether ptr is currently active in the arena. */
JEMALLOC_ALWAYS_INLINE bool
arena_tcache_dalloc_small_safety_check(tsdn_t *tsdn, void *ptr) {
if (!config_debug) {
return false;
}
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global, ptr);
szind_t binind = edata_szind_get(edata);
div_info_t div_info = arena_binind_div_info[binind];
/*
* Calls the internal function arena_slab_regind_impl because the
* safety check does not require a lock.
*/
size_t regind = arena_slab_regind_impl(&div_info, binind, edata, ptr);
slab_data_t *slab_data = edata_slab_data_get(edata);
const bin_info_t *bin_info = &bin_infos[binind];
assert(edata_nfree_get(edata) < bin_info->nregs);
if (unlikely(!bitmap_get(slab_data->bitmap, &bin_info->bitmap_info,
regind))) {
safety_check_fail(
"Invalid deallocation detected: the pointer being freed (%p) not "
"currently active, possibly caused by double free bugs.\n", ptr);
return true;
}
return false;
}
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
JEMALLOC_ALWAYS_INLINE void
Pass dealloc_ctx down free() fast path. This gets rid of the redundent rtree lookup down fast path.
2017-04-08 05:12:30 +08:00
arena_dalloc(tsdn_t *tsdn, void *ptr, tcache_t *tcache,
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t *caller_alloc_ctx, bool slow_path) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
assert(!tsdn_null(tsdn) || tcache == NULL);
assert(ptr != NULL);
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
if (unlikely(tcache == NULL)) {
arena_dalloc_no_tcache(tsdn, ptr);
return;
}
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t alloc_ctx;
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (caller_alloc_ctx != NULL) {
alloc_ctx = *caller_alloc_ctx;
Pass dealloc_ctx down free() fast path. This gets rid of the redundent rtree lookup down fast path.
2017-04-08 05:12:30 +08:00
} else {
Use compiler-provided assume builtins when available There are several benefits to this: 1. It's cleaner and more reliable to use the builtin to inform the compiler of assumptions instead of hoping that the optimizer understands your intentions. 2. `clang` will warn you if any of your assumptions would produce side-effects (which the compiler will discard). [This blog post](https://fastcompression.blogspot.com/2019/01/compiler-checked-contracts.html) by Yann Collet highlights that a hazard of using the `unreachable()`-based method of signaling assumptions is that it can sometimes result in additional instructions being generated (see [this Godbolt link](https://godbolt.org/z/lKNMs3) from the blog post for an example).
2023-07-28 02:49:07 +08:00
util_assume(tsdn != NULL);
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
emap_alloc_ctx_lookup(tsdn, &arena_emap_global, ptr,
&alloc_ctx);
Pass dealloc_ctx down free() fast path. This gets rid of the redundent rtree lookup down fast path.
2017-04-08 05:12:30 +08:00
}
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
if (config_debug) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global,
ptr);
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(alloc_ctx.szind == edata_szind_get(edata));
assert(alloc_ctx.szind < SC_NSIZES);
assert(alloc_ctx.slab == edata_slab_get(edata));
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (likely(alloc_ctx.slab)) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
/* Small allocation. */
Add double free detection using slab bitmap for debug build Add a sanity check for double free issue in the arena in case that the tcache has been flushed.
2022-08-10 07:39:02 +08:00
if (arena_tcache_dalloc_small_safety_check(tsdn, ptr)) {
return;
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
tcache_dalloc_small(tsdn_tsd(tsdn), tcache, ptr,
alloc_ctx.szind, slow_path);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
} else {
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
arena_dalloc_large(tsdn, ptr, tcache, alloc_ctx.szind,
slow_path);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
}
Get rid of most of the various inline macros.
2017-04-22 00:37:34 +08:00
static inline void
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
arena_sdalloc_no_tcache(tsdn_t *tsdn, void *ptr, size_t size) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
assert(ptr != NULL);
SC: Make some key size classes static. The largest small class, smallest large class, and largest large class may all be needed down fast paths; to avoid the risk of touching another cache line, we can make them available as constants.
2018-07-12 07:05:58 +08:00
assert(size <= SC_LARGE_MAXCLASS);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t alloc_ctx;
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
if (!config_prof || !opt_prof) {
/*
* There is no risk of being confused by a promoted sampled
* object, so base szind and slab on the given size.
*/
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
alloc_ctx.szind = sz_size2index(size);
alloc_ctx.slab = (alloc_ctx.szind < SC_NBINS);
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
}
if ((config_prof && opt_prof) || config_debug) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
emap_alloc_ctx_lookup(tsdn, &arena_emap_global, ptr,
&alloc_ctx);
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(alloc_ctx.szind == sz_size2index(size));
assert((config_prof && opt_prof)
|| alloc_ctx.slab == (alloc_ctx.szind < SC_NBINS));
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
if (config_debug) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn,
&arena_emap_global, ptr);
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(alloc_ctx.szind == edata_szind_get(edata));
assert(alloc_ctx.slab == edata_slab_get(edata));
Remove extent dereferences from the deallocation fast paths.
2017-03-17 17:45:12 +08:00
}
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (likely(alloc_ctx.slab)) {
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
/* Small allocation. */
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
arena_dalloc_small(tsdn, ptr);
} else {
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
arena_dalloc_large_no_tcache(tsdn, ptr, alloc_ctx.szind);
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
}
}
JEMALLOC_ALWAYS_INLINE void
arena_sdalloc(tsdn_t *tsdn, void *ptr, size_t size, tcache_t *tcache,
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t *caller_alloc_ctx, bool slow_path) {
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
assert(!tsdn_null(tsdn) || tcache == NULL);
assert(ptr != NULL);
SC: Make some key size classes static. The largest small class, smallest large class, and largest large class may all be needed down fast paths; to avoid the risk of touching another cache line, we can make them available as constants.
2018-07-12 07:05:58 +08:00
assert(size <= SC_LARGE_MAXCLASS);
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
if (unlikely(tcache == NULL)) {
arena_sdalloc_no_tcache(tsdn, ptr, size);
return;
}
Emap: Standardize naming. Namespace everything under emap_, always specify what it is we're looking up (emap_lookup -> emap_edata_lookup), and use "ctx" over "info".
2020-02-07 05:45:04 +08:00
emap_alloc_ctx_t alloc_ctx;
Pass dalloc_ctx down the sdalloc path. This avoids redundant rtree lookups.
2017-04-12 05:56:43 +08:00
if (config_prof && opt_prof) {
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (caller_alloc_ctx == NULL) {
Pass dalloc_ctx down the sdalloc path. This avoids redundant rtree lookups.
2017-04-12 05:56:43 +08:00
/* Uncommon case and should be a static check. */
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
emap_alloc_ctx_lookup(tsdn, &arena_emap_global, ptr,
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
&alloc_ctx);
assert(alloc_ctx.szind == sz_size2index(size));
} else {
alloc_ctx = *caller_alloc_ctx;
Pass dalloc_ctx down the sdalloc path. This avoids redundant rtree lookups.
2017-04-12 05:56:43 +08:00
}
} else {
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
/*
* There is no risk of being confused by a promoted sampled
* object, so base szind and slab on the given size.
*/
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
alloc_ctx.szind = sz_size2index(size);
alloc_ctx.slab = (alloc_ctx.szind < SC_NBINS);
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
}
Pass dalloc_ctx down the sdalloc path. This avoids redundant rtree lookups.
2017-04-12 05:56:43 +08:00
if (config_debug) {
PA: Parameterize emap. Move emap_global to arena. This lets us test the PA module without interfering with the global emap used by the real allocator (the one not under test).
2020-03-15 01:49:34 +08:00
edata_t *edata = emap_edata_lookup(tsdn, &arena_emap_global,
ptr);
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
assert(alloc_ctx.szind == edata_szind_get(edata));
assert(alloc_ctx.slab == edata_slab_get(edata));
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
if (likely(alloc_ctx.slab)) {
Further specialize arena_[s]dalloc() tcache fast path. Use tsd_rtree_ctx() rather than tsdn_rtree_ctx() when tcache is non-NULL, in order to avoid an extra branch (and potentially extra stack space) in the fast path.
2017-03-23 02:00:40 +08:00
/* Small allocation. */
Add double free detection using slab bitmap for debug build Add a sanity check for double free issue in the arena in case that the tcache has been flushed.
2022-08-10 07:39:02 +08:00
if (arena_tcache_dalloc_small_safety_check(tsdn, ptr)) {
return;
}
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
tcache_dalloc_small(tsdn_tsd(tsdn), tcache, ptr,
alloc_ctx.szind, slow_path);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
} else {
Emap: Remove direct access to emap internals. In the process, we do a few local cleanups and optimizations. In particular, the size safety check on tcache flush no longer does a redundant load.
2020-02-07 05:16:07 +08:00
arena_dalloc_large(tsdn, ptr, tcache, alloc_ctx.szind,
slow_path);
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
}
}
Move cache index randomization out of extent. This is logically at a higher level of the stack; extent should just allocate things at the page-level; it shouldn't care exactly why the callers wants a given number of pages.
2020-03-09 01:11:02 +08:00
static inline void
arena_cache_oblivious_randomize(tsdn_t *tsdn, arena_t *arena, edata_t *edata,
size_t alignment) {
assert(edata_base_get(edata) == edata_addr_get(edata));
if (alignment < PAGE) {
unsigned lg_range = LG_PAGE -
lg_floor(CACHELINE_CEILING(alignment));
size_t r;
if (!tsdn_null(tsdn)) {
tsd_t *tsd = tsdn_tsd(tsdn);
r = (size_t)prng_lg_range_u64(
tsd_prng_statep_get(tsd), lg_range);
} else {
uint64_t stack_value = (uint64_t)(uintptr_t)&r;
r = (size_t)prng_lg_range_u64(&stack_value, lg_range);
}
uintptr_t random_offset = ((uintptr_t)r) << (LG_PAGE -
lg_range);
Fix all optimization-inhibiting integer-to-pointer casts Following from PR #2481, we replace all integer-to-pointer casts [which hide pointer provenance information (and thus inhibit optimizations)](https://clang.llvm.org/extra/clang-tidy/checks/performance/no-int-to-ptr.html) with equivalent operations that preserve this information. I have enabled the corresponding clang-tidy check in our static analysis CI so that we do not get bitten by this again in the future.
2023-07-25 01:33:36 +08:00
edata->e_addr = (void *)((byte_t *)edata->e_addr +
Move cache index randomization out of extent. This is logically at a higher level of the stack; extent should just allocate things at the page-level; it shouldn't care exactly why the callers wants a given number of pages.
2020-03-09 01:11:02 +08:00
random_offset);
assert(ALIGNMENT_ADDR2BASE(edata->e_addr, alignment) ==
edata->e_addr);
}
}
Tcache flush: keep common path state in registers. By carefully force-inlining the division constants and the operation sum count, we can eliminate redundant operations in the arena-level dalloc function. Do so.
2021-01-30 08:06:28 +08:00
/*
* The dalloc bin info contains just the information that the common paths need
* during tcache flushes. By force-inlining these paths, and using local copies
* of data (so that the compiler knows it's constant), we avoid a whole bunch of
* redundant loads and stores by leaving this information in registers.
*/
typedef struct arena_dalloc_bin_locked_info_s arena_dalloc_bin_locked_info_t;
struct arena_dalloc_bin_locked_info_s {
div_info_t div_info;
uint32_t nregs;
uint64_t ndalloc;
};
JEMALLOC_ALWAYS_INLINE size_t
arena_slab_regind(arena_dalloc_bin_locked_info_t *info, szind_t binind,
edata_t *slab, const void *ptr) {
Add double free detection using slab bitmap for debug build Add a sanity check for double free issue in the arena in case that the tcache has been flushed.
2022-08-10 07:39:02 +08:00
size_t regind = arena_slab_regind_impl(&info->div_info, binind, slab, ptr);
Tcache flush: keep common path state in registers. By carefully force-inlining the division constants and the operation sum count, we can eliminate redundant operations in the arena-level dalloc function. Do so.
2021-01-30 08:06:28 +08:00
return regind;
}
JEMALLOC_ALWAYS_INLINE void
arena_dalloc_bin_locked_begin(arena_dalloc_bin_locked_info_t *info,
szind_t binind) {
info->div_info = arena_binind_div_info[binind];
info->nregs = bin_infos[binind].nregs;
info->ndalloc = 0;
}
/*
* Does the deallocation work associated with freeing a single pointer (a
* "step") in between a arena_dalloc_bin_locked begin and end call.
*
* Returns true if arena_slab_dalloc must be called on slab. Doesn't do
* stats updates, which happen during finish (this lets running counts get left
* in a register).
*/
JEMALLOC_ALWAYS_INLINE bool
arena_dalloc_bin_locked_step(tsdn_t *tsdn, arena_t *arena, bin_t *bin,
arena_dalloc_bin_locked_info_t *info, szind_t binind, edata_t *slab,
void *ptr) {
const bin_info_t *bin_info = &bin_infos[binind];
size_t regind = arena_slab_regind(info, binind, slab, ptr);
slab_data_t *slab_data = edata_slab_data_get(slab);
assert(edata_nfree_get(slab) < bin_info->nregs);
/* Freeing an unallocated pointer can cause assertion failure. */
assert(bitmap_get(slab_data->bitmap, &bin_info->bitmap_info, regind));
bitmap_unset(slab_data->bitmap, &bin_info->bitmap_info, regind);
edata_nfree_inc(slab);
if (config_stats) {
info->ndalloc++;
}
unsigned nfree = edata_nfree_get(slab);
if (nfree == bin_info->nregs) {
arena_dalloc_bin_locked_handle_newly_empty(tsdn, arena, slab,
bin);
return true;
} else if (nfree == 1 && slab != bin->slabcur) {
arena_dalloc_bin_locked_handle_newly_nonempty(tsdn, arena, slab,
bin);
}
return false;
}
JEMALLOC_ALWAYS_INLINE void
arena_dalloc_bin_locked_finish(tsdn_t *tsdn, arena_t *arena, bin_t *bin,
arena_dalloc_bin_locked_info_t *info) {
if (config_stats) {
bin->stats.ndalloc += info->ndalloc;
assert(bin->stats.curregs >= (size_t)info->ndalloc);
bin->stats.curregs -= (size_t)info->ndalloc;
}
}
Arena: share bin offsets in a global. This saves us a cache miss when lookup up the arena bin offset in a remote arena during tcache flush. All arenas share the base offset, and so we don't need to look it up repeatedly for each arena. Secondarily, it shaves 288 bytes off the arena on, e.g., x86-64.
2021-01-31 07:35:33 +08:00
static inline bin_t *
arena_get_bin(arena_t *arena, szind_t binind, unsigned binshard) {
Fix all optimization-inhibiting integer-to-pointer casts Following from PR #2481, we replace all integer-to-pointer casts [which hide pointer provenance information (and thus inhibit optimizations)](https://clang.llvm.org/extra/clang-tidy/checks/performance/no-int-to-ptr.html) with equivalent operations that preserve this information. I have enabled the corresponding clang-tidy check in our static analysis CI so that we do not get bitten by this again in the future.
2023-07-25 01:33:36 +08:00
bin_t *shard0 = (bin_t *)((byte_t *)arena + arena_bin_offsets[binind]);
Arena: share bin offsets in a global. This saves us a cache miss when lookup up the arena bin offset in a remote arena during tcache flush. All arenas share the base offset, and so we don't need to look it up repeatedly for each arena. Secondarily, it shaves 288 bytes off the arena on, e.g., x86-64.
2021-01-31 07:35:33 +08:00
return shard0 + binshard;
}
Break up headers into constituent parts This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
2017-01-11 10:06:31 +08:00
#endif /* JEMALLOC_INTERNAL_ARENA_INLINES_B_H */
Reference in New Issue Copy Permalink