project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
94c5d22a4d
server-skynet-source-3rd-je.../include/jemalloc/internal/extent.h

500 lines
13 KiB
C
Raw Normal View History

Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
typedef struct extent_s extent_t;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
#define EXTENT_HOOKS_INITIALIZER NULL
s/CHUNK_HOOKS_INITIALIZER/EXTENT_HOOKS_INITIALIZER/g
2016-06-02 02:56:45 +08:00
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
/* Extent (span of pages). Use accessor functions for e_* fields. */
struct extent_s {
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
/* Arena from which this extent came, if any. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
arena_t *e_arena;
Implement allocation profiling and leack checking. Add the --enable-prof and --enable-prof-libunwind configure options. Add the B/b, F/f, I/i, L/l, and U/u JEMALLOC_OPTIONS. Interval-based profile dump triggering is not yet implemented. Add supporting generic code: * Add memory barriers. * Add prn (LCG PRNG). * Add hash (Murmur hash function). * Add ckh (cuckoo hash tables).
2010-02-11 02:37:56 +08:00
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
/* Pointer to the extent that this structure is responsible for. */
void *e_addr;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
/* Extent size. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
size_t e_size;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
/*
* Usable size, typically smaller than extent size due to large_pad or
* promotion of sampled small regions.
*/
size_t e_usize;
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
/*
* Serial number (potentially non-unique).
*
* In principle serial numbers can wrap around on 32-bit systems if
* JEMALLOC_MUNMAP is defined, but as long as comparison functions fall
* back on address comparison for equal serial numbers, stable (if
* imperfect) ordering is maintained.
*
* Serial numbers may not be unique even in the absence of wrap-around,
* e.g. when splitting an extent and assigning the same serial number to
* both resulting adjacent extents.
*/
size_t e_sn;
Add extent_active_[gs]et(). Always initialize extents' runs_dirty and chunks_cache linkage.
2016-03-28 18:17:10 +08:00
/* True if extent is active (in use). */
bool e_active;
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
/*
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
* The zeroed flag is used by extent recycling code to track whether
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
* memory is zero-filled.
*/
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
bool e_zeroed;
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
/*
* True if physical memory is committed to the extent, whether
* explicitly or implicitly as on a system that overcommits and
Arena chunk decommit cleanups and fixes. Decommit arena chunk header during chunk deallocation if the rest of the chunk is decommitted.
2015-08-10 07:47:27 +08:00
* satisfies physical memory needs on demand via soft page faults.
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
*/
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
bool e_committed;
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
Reduce extent_node_t size to fit in one cache line.
2015-01-31 13:21:16 +08:00
/*
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
* The slab flag indicates whether the extent is used for a slab of
* small regions. This helps differentiate small size classes, and it
* indicates whether interior pointers can be looked up via iealloc().
Reduce extent_node_t size to fit in one cache line.
2015-01-31 13:21:16 +08:00
*/
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
bool e_slab;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
Make extent_prof_tctx_[gs]et() atomic.
2016-05-25 12:13:36 +08:00
union {
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
/* Small region slab metadata. */
arena_slab_data_t e_slab_data;
Rename huge to large.
2016-06-01 05:50:21 +08:00
/* Profile counters, used for large objects. */
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
union {
void *e_prof_tctx_pun;
prof_tctx_t *e_prof_tctx;
};
Make extent_prof_tctx_[gs]et() atomic.
2016-05-25 12:13:36 +08:00
};
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
/*
* Linkage for arena's extents_dirty and arena_bin_t's slabs_full rings.
*/
qr(extent_t) qr_link;
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
union {
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
/* Linkage for per size class sn/address-ordered heaps. */
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
phn(extent_t) ph_link;
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
Rename huge to large.
2016-06-01 05:50:21 +08:00
/* Linkage for arena's large and extent_cache lists. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
ql_elm(extent_t) ql_link;
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
};
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
};
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
typedef ph(extent_t) extent_heap_t;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extern rtree_t extents_rtree;
extern const extent_hooks_t extent_hooks_default;
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
extent_t *extent_alloc(tsdn_t *tsdn, arena_t *arena);
void extent_dalloc(tsdn_t *tsdn, arena_t *arena, extent_t *extent);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t *extent_hooks_get(arena_t *arena);
extent_hooks_t *extent_hooks_set(arena_t *arena, extent_hooks_t *extent_hooks);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
Replace extent_tree_szad_* with extent_heap_*.
2016-05-18 05:58:56 +08:00
#ifdef JEMALLOC_JET
typedef size_t (extent_size_quantize_t)(size_t);
extern extent_size_quantize_t *extent_size_quantize_floor;
extern extent_size_quantize_t *extent_size_quantize_ceil;
#else
size_t extent_size_quantize_floor(size_t size);
size_t extent_size_quantize_ceil(size_t size);
#endif
ph_proto(, extent_heap_, extent_heap_t, extent_t)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Protect extents_dirty access with extents_mtx. This fixes race conditions during purging.
2016-09-23 02:57:28 +08:00
extent_t *extent_alloc_cache_locked(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
size_t alignment, bool *zero, bool *commit, bool slab);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_t *extent_alloc_cache(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
size_t alignment, bool *zero, bool *commit, bool slab);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_t *extent_alloc_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, void *new_addr, size_t usize, size_t pad,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t alignment, bool *zero, bool *commit, bool slab);
Fix extent_alloc_dss() regressions. Page-align the gap, if any, and add/use extent_dalloc_gap(), which registers the gap extent before deallocation.
2016-06-06 06:27:20 +08:00
void extent_dalloc_gap(tsdn_t *tsdn, arena_t *arena, extent_t *extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
void extent_dalloc_cache(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent);
Implement arena.<i>.destroy . Add MALLCTL_ARENAS_DESTROYED for accessing destroyed arena stats as an analogue to MALLCTL_ARENAS_ALL. This resolves #382.
2017-01-04 09:21:59 +08:00
bool extent_dalloc_wrapper_try(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
void extent_dalloc_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
bool extent_commit_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t length);
bool extent_decommit_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t length);
Refactor purging and splitting/merging. Split purging into lazy and forced variants. Use the forced variant for zeroing dss. Add support for NULL function pointers as an opt-out mechanism for the dalloc, commit, decommit, purge_lazy, purge_forced, split, and merge fields of extent_hooks_t. Add short-circuiting checks in large_ralloc_no_move_{shrink,expand}() so that no attempt is made if splitting/merging is not supported. This resolves #268.
2016-12-04 07:38:25 +08:00
bool extent_purge_lazy_wrapper(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
size_t length);
bool extent_purge_forced_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t offset,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t length);
extent_t *extent_split_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *extent, size_t size_a,
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
size_t usize_a, size_t size_b, size_t usize_b);
bool extent_merge_wrapper(tsdn_t *tsdn, arena_t *arena,
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t **r_extent_hooks, extent_t *a, extent_t *b);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
bool extent_boot(void);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
#ifndef JEMALLOC_ENABLE_INLINE
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_t *extent_lookup(tsdn_t *tsdn, const void *ptr, bool dependent);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
arena_t *extent_arena_get(const extent_t *extent);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
void *extent_base_get(const extent_t *extent);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
void *extent_addr_get(const extent_t *extent);
size_t extent_size_get(const extent_t *extent);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
size_t extent_usize_get(const extent_t *extent);
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
void *extent_before_get(const extent_t *extent);
void *extent_last_get(const extent_t *extent);
Add extent_past_get().
2016-04-07 22:34:26 +08:00
void *extent_past_get(const extent_t *extent);
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
size_t extent_sn_get(const extent_t *extent);
Add extent_active_[gs]et(). Always initialize extents' runs_dirty and chunks_cache linkage.
2016-03-28 18:17:10 +08:00
bool extent_active_get(const extent_t *extent);
Use rtree rather than [sz]ad trees for chunk split/coalesce operations.
2016-05-17 04:25:18 +08:00
bool extent_retained_get(const extent_t *extent);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
bool extent_zeroed_get(const extent_t *extent);
bool extent_committed_get(const extent_t *extent);
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
bool extent_slab_get(const extent_t *extent);
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
arena_slab_data_t *extent_slab_data_get(extent_t *extent);
const arena_slab_data_t *extent_slab_data_get_const(const extent_t *extent);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
prof_tctx_t *extent_prof_tctx_get(const extent_t *extent);
void extent_arena_set(extent_t *extent, arena_t *arena);
void extent_addr_set(extent_t *extent, void *addr);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
void extent_addr_randomize(tsdn_t *tsdn, extent_t *extent, size_t alignment);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
void extent_size_set(extent_t *extent, size_t size);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
void extent_usize_set(extent_t *extent, size_t usize);
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
void extent_sn_set(extent_t *extent, size_t sn);
Add extent_active_[gs]et(). Always initialize extents' runs_dirty and chunks_cache linkage.
2016-03-28 18:17:10 +08:00
void extent_active_set(extent_t *extent, bool active);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
void extent_zeroed_set(extent_t *extent, bool zeroed);
void extent_committed_set(extent_t *extent, bool committed);
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
void extent_slab_set(extent_t *extent, bool slab);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
void extent_prof_tctx_set(extent_t *extent, prof_tctx_t *tctx);
void extent_init(extent_t *extent, arena_t *arena, void *addr,
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
size_t size, size_t usize, size_t sn, bool active, bool zeroed,
bool committed, bool slab);
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
void extent_ring_insert(extent_t *sentinel, extent_t *extent);
void extent_ring_remove(extent_t *extent);
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
int extent_sn_comp(const extent_t *a, const extent_t *b);
int extent_ad_comp(const extent_t *a, const extent_t *b);
int extent_snad_comp(const extent_t *a, const extent_t *b);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
#endif
#if (defined(JEMALLOC_ENABLE_INLINE) || defined(JEMALLOC_EXTENT_C_))
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
JEMALLOC_INLINE extent_t *
extent_lookup(tsdn_t *tsdn, const void *ptr, bool dependent)
{
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
return (rtree_read(tsdn, &extents_rtree, rtree_ctx, (uintptr_t)ptr,
dependent));
s/chunk_lookup/extent_lookup/g, s/chunks_rtree/extents_rtree/g
2016-06-02 03:10:39 +08:00
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE arena_t *
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_arena_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_arena);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
JEMALLOC_INLINE void *
extent_base_get(const extent_t *extent)
{
assert(extent->e_addr == PAGE_ADDR2BASE(extent->e_addr) ||
!extent->e_slab);
return (PAGE_ADDR2BASE(extent->e_addr));
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE void *
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_addr_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
assert(extent->e_addr == PAGE_ADDR2BASE(extent->e_addr) ||
!extent->e_slab);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_addr);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE size_t
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_size_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_size);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
JEMALLOC_INLINE size_t
extent_usize_get(const extent_t *extent)
{
assert(!extent->e_slab);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
return (extent->e_usize);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
}
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
JEMALLOC_INLINE void *
extent_before_get(const extent_t *extent)
{
Fix extent_{before,last,past}() to return page-aligned results.
2016-09-24 03:17:42 +08:00
return ((void *)((uintptr_t)extent_base_get(extent) - PAGE));
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
}
JEMALLOC_INLINE void *
extent_last_get(const extent_t *extent)
{
Fix extent_{before,last,past}() to return page-aligned results.
2016-09-24 03:17:42 +08:00
return ((void *)((uintptr_t)extent_base_get(extent) +
extent_size_get(extent) - PAGE));
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
}
Add extent_past_get().
2016-04-07 22:34:26 +08:00
JEMALLOC_INLINE void *
extent_past_get(const extent_t *extent)
{
Fix extent_{before,last,past}() to return page-aligned results.
2016-09-24 03:17:42 +08:00
return ((void *)((uintptr_t)extent_base_get(extent) +
extent_size_get(extent)));
Add extent_past_get().
2016-04-07 22:34:26 +08:00
}
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
JEMALLOC_INLINE size_t
extent_sn_get(const extent_t *extent)
{
return (extent->e_sn);
}
Add extent_active_[gs]et(). Always initialize extents' runs_dirty and chunks_cache linkage.
2016-03-28 18:17:10 +08:00
JEMALLOC_INLINE bool
extent_active_get(const extent_t *extent)
{
return (extent->e_active);
}
Use rtree rather than [sz]ad trees for chunk split/coalesce operations.
2016-05-17 04:25:18 +08:00
JEMALLOC_INLINE bool
extent_retained_get(const extent_t *extent)
{
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
return (qr_next(extent, qr_link) == extent);
Use rtree rather than [sz]ad trees for chunk split/coalesce operations.
2016-05-17 04:25:18 +08:00
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
JEMALLOC_INLINE bool
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_zeroed_get(const extent_t *extent)
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_zeroed);
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE bool
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_committed_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_committed);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE bool
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
extent_slab_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
return (extent->e_slab);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
JEMALLOC_INLINE arena_slab_data_t *
extent_slab_data_get(extent_t *extent)
{
assert(extent->e_slab);
return (&extent->e_slab_data);
}
JEMALLOC_INLINE const arena_slab_data_t *
extent_slab_data_get_const(const extent_t *extent)
{
assert(extent->e_slab);
return (&extent->e_slab_data);
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE prof_tctx_t *
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Make extent_prof_tctx_[gs]et() atomic.
2016-05-25 12:13:36 +08:00
return ((prof_tctx_t *)atomic_read_p(
&((extent_t *)extent)->e_prof_tctx_pun));
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_arena_set(extent_t *extent, arena_t *arena)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_arena = arena;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_addr_set(extent_t *extent, void *addr)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_addr = addr;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
JEMALLOC_INLINE void
extent_addr_randomize(tsdn_t *tsdn, extent_t *extent, size_t alignment)
{
assert(extent_base_get(extent) == extent_addr_get(extent));
if (alignment < PAGE) {
unsigned lg_range = LG_PAGE -
lg_floor(CACHELINE_CEILING(alignment));
Refactor prng to not use 64-bit atomics on 32-bit platforms. This resolves #495.
2016-11-08 02:52:44 +08:00
size_t r =
prng_lg_range_zu(&extent_arena_get(extent)->offset_state,
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
lg_range, true);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
uintptr_t random_offset = ((uintptr_t)r) << (LG_PAGE -
lg_range);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
extent->e_addr = (void *)((uintptr_t)extent->e_addr +
random_offset);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
assert(ALIGNMENT_ADDR2BASE(extent->e_addr, alignment) ==
extent->e_addr);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
}
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_size_set(extent_t *extent, size_t size)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_size = size;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
JEMALLOC_INLINE void
extent_usize_set(extent_t *extent, size_t usize)
{
extent->e_usize = usize;
}
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
JEMALLOC_INLINE void
extent_sn_set(extent_t *extent, size_t sn)
{
extent->e_sn = sn;
}
Add extent_active_[gs]et(). Always initialize extents' runs_dirty and chunks_cache linkage.
2016-03-28 18:17:10 +08:00
JEMALLOC_INLINE void
extent_active_set(extent_t *extent, bool active)
{
extent->e_active = active;
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_zeroed_set(extent_t *extent, bool zeroed)
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_zeroed = zeroed;
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_committed_set(extent_t *extent, bool committed)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_committed = committed;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
extent_slab_set(extent_t *extent, bool slab)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
extent->e_slab = slab;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_set(extent_t *extent, prof_tctx_t *tctx)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Make extent_prof_tctx_[gs]et() atomic.
2016-05-25 12:13:36 +08:00
atomic_write_p(&extent->e_prof_tctx_pun, tctx);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_init(extent_t *extent, arena_t *arena, void *addr, size_t size,
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
size_t usize, size_t sn, bool active, bool zeroed, bool committed,
bool slab)
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
{
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
assert(addr == PAGE_ADDR2BASE(addr) || !slab);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_arena_set(extent, arena);
extent_addr_set(extent, addr);
extent_size_set(extent, size);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
extent_usize_set(extent, usize);
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
extent_sn_set(extent, sn);
Add extent_active_[gs]et(). Always initialize extents' runs_dirty and chunks_cache linkage.
2016-03-28 18:17:10 +08:00
extent_active_set(extent, active);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_zeroed_set(extent, zeroed);
extent_committed_set(extent, committed);
Replace extent_achunk_[gs]et() with extent_slab_[gs]et().
2016-04-07 22:24:14 +08:00
extent_slab_set(extent, slab);
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
if (config_prof)
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_set(extent, NULL);
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
qr_new(extent, qr_link);
arena_chunk_dirty_node_init() --> extent_node_dirty_linkage_init()
2015-02-18 14:23:10 +08:00
}
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
JEMALLOC_INLINE void
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
extent_ring_insert(extent_t *sentinel, extent_t *extent)
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
{
Add a_type parameter to qr_{meld,split}().
2016-12-13 10:04:20 +08:00
qr_meld(sentinel, extent, extent_t, qr_link);
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
}
JEMALLOC_INLINE void
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
extent_ring_remove(extent_t *extent)
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
{
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
qr_remove(extent, qr_link);
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
}
Add extent serial numbers. Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This resolves #147.
2016-11-16 05:07:53 +08:00
JEMALLOC_INLINE int
extent_sn_comp(const extent_t *a, const extent_t *b)
{
size_t a_sn = extent_sn_get(a);
size_t b_sn = extent_sn_get(b);
return ((a_sn > b_sn) - (a_sn < b_sn));
}
JEMALLOC_INLINE int
extent_ad_comp(const extent_t *a, const extent_t *b)
{
uintptr_t a_addr = (uintptr_t)extent_addr_get(a);
uintptr_t b_addr = (uintptr_t)extent_addr_get(b);
return ((a_addr > b_addr) - (a_addr < b_addr));
}
JEMALLOC_INLINE int
extent_snad_comp(const extent_t *a, const extent_t *b)
{
int ret;
ret = extent_sn_comp(a, b);
if (ret != 0)
return (ret);
ret = extent_ad_comp(a, b);
return (ret);
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
#endif
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
#include "jemalloc/internal/extent_dss.h"
#include "jemalloc/internal/extent_mmap.h"
Reference in New Issue Copy Permalink