project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f4a58847d3
server-skynet-source-3rd-je.../include/jemalloc/internal/extent.h

239 lines
5.4 KiB
C
Raw Normal View History

Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
typedef struct extent_s extent_t;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
/* Extent (span of pages). Use accessor functions for e_* fields. */
struct extent_s {
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
/* Arena from which this extent came, if any. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
arena_t *e_arena;
Implement allocation profiling and leack checking. Add the --enable-prof and --enable-prof-libunwind configure options. Add the B/b, F/f, I/i, L/l, and U/u JEMALLOC_OPTIONS. Interval-based profile dump triggering is not yet implemented. Add supporting generic code: * Add memory barriers. * Add prn (LCG PRNG). * Add hash (Murmur hash function). * Add ckh (cuckoo hash tables).
2010-02-11 02:37:56 +08:00
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
/* Pointer to the extent that this structure is responsible for. */
void *e_addr;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
/* Total region size. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
size_t e_size;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
/*
* The zeroed flag is used by chunk recycling code to track whether
* memory is zero-filled.
*/
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
bool e_zeroed;
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
/*
* True if physical memory is committed to the extent, whether
* explicitly or implicitly as on a system that overcommits and
Arena chunk decommit cleanups and fixes. Decommit arena chunk header during chunk deallocation if the rest of the chunk is decommitted.
2015-08-10 07:47:27 +08:00
* satisfies physical memory needs on demand via soft page faults.
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
*/
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
bool e_committed;
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
Reduce extent_node_t size to fit in one cache line.
2015-01-31 13:21:16 +08:00
/*
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
* The achunk flag is used to validate that huge allocation lookups
* don't return arena chunks.
Reduce extent_node_t size to fit in one cache line.
2015-01-31 13:21:16 +08:00
*/
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
bool e_achunk;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
/* Profile counters, used for huge objects. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
prof_tctx_t *e_prof_tctx;
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
/* Linkage for arena's runs_dirty and chunks_cache rings. */
Normalize rdelm/rd structure field naming.
2015-03-11 09:29:49 +08:00
arena_runs_dirty_link_t rd;
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
qr(extent_t) cc_link;
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
union {
/* Linkage for the size/address-ordered tree. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
rb_node(extent_t) szad_link;
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
Implement the arena.<i>.reset mallctl. This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
2016-04-23 05:37:17 +08:00
/* Linkage for arena's achunks, huge, and node_cache lists. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
ql_elm(extent_t) ql_link;
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
};
/* Linkage for the address-ordered tree. */
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
rb_node(extent_t) ad_link;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
};
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
typedef rb_tree(extent_t) extent_tree_t;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
rb_proto(, extent_tree_szad_, extent_tree_t, extent_t)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
rb_proto(, extent_tree_ad_, extent_tree_t, extent_t)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
#ifndef JEMALLOC_ENABLE_INLINE
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
arena_t *extent_arena_get(const extent_t *extent);
void *extent_addr_get(const extent_t *extent);
size_t extent_size_get(const extent_t *extent);
bool extent_zeroed_get(const extent_t *extent);
bool extent_committed_get(const extent_t *extent);
bool extent_achunk_get(const extent_t *extent);
prof_tctx_t *extent_prof_tctx_get(const extent_t *extent);
void extent_arena_set(extent_t *extent, arena_t *arena);
void extent_addr_set(extent_t *extent, void *addr);
void extent_size_set(extent_t *extent, size_t size);
void extent_zeroed_set(extent_t *extent, bool zeroed);
void extent_committed_set(extent_t *extent, bool committed);
void extent_achunk_set(extent_t *extent, bool achunk);
void extent_prof_tctx_set(extent_t *extent, prof_tctx_t *tctx);
void extent_init(extent_t *extent, arena_t *arena, void *addr,
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
size_t size, bool zeroed, bool committed);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
void extent_dirty_linkage_init(extent_t *extent);
void extent_dirty_insert(extent_t *extent,
arena_runs_dirty_link_t *runs_dirty, extent_t *chunks_dirty);
void extent_dirty_remove(extent_t *extent);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
#endif
#if (defined(JEMALLOC_ENABLE_INLINE) || defined(JEMALLOC_EXTENT_C_))
JEMALLOC_INLINE arena_t *
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_arena_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_arena);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void *
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_addr_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_addr);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE size_t
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_size_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_size);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
JEMALLOC_INLINE bool
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_zeroed_get(const extent_t *extent)
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_zeroed);
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE bool
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_committed_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
assert(!extent->e_achunk);
return (extent->e_committed);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE bool
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_achunk_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_achunk);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE prof_tctx_t *
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_get(const extent_t *extent)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
return (extent->e_prof_tctx);
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_arena_set(extent_t *extent, arena_t *arena)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_arena = arena;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_addr_set(extent_t *extent, void *addr)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_addr = addr;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_size_set(extent_t *extent, size_t size)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_size = size;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_zeroed_set(extent_t *extent, bool zeroed)
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_zeroed = zeroed;
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_committed_set(extent_t *extent, bool committed)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_committed = committed;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_achunk_set(extent_t *extent, bool achunk)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_achunk = achunk;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_set(extent_t *extent, prof_tctx_t *tctx)
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent->e_prof_tctx = tctx;
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
}
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_init(extent_t *extent, arena_t *arena, void *addr, size_t size,
Implement chunk hook support for page run commit/decommit. Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
2015-08-05 01:49:46 +08:00
bool zeroed, bool committed)
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_arena_set(extent, arena);
extent_addr_set(extent, addr);
extent_size_set(extent, size);
extent_zeroed_set(extent, zeroed);
extent_committed_set(extent, committed);
extent_achunk_set(extent, false);
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
if (config_prof)
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_set(extent, NULL);
Simplify extent_node_t and add extent_node_init().
2015-02-18 07:13:52 +08:00
}
arena_chunk_dirty_node_init() --> extent_node_dirty_linkage_init()
2015-02-18 14:23:10 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_dirty_linkage_init(extent_t *extent)
arena_chunk_dirty_node_init() --> extent_node_dirty_linkage_init()
2015-02-18 14:23:10 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
qr_new(&extent->rd, rd_link);
qr_new(extent, cc_link);
arena_chunk_dirty_node_init() --> extent_node_dirty_linkage_init()
2015-02-18 14:23:10 +08:00
}
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_dirty_insert(extent_t *extent,
arena_runs_dirty_link_t *runs_dirty, extent_t *chunks_dirty)
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
qr_meld(runs_dirty, &extent->rd, rd_link);
qr_meld(chunks_dirty, extent, cc_link);
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
}
JEMALLOC_INLINE void
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_dirty_remove(extent_t *extent)
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
qr_remove(&extent->rd, rd_link);
qr_remove(extent, cc_link);
Rename "dirty chunks" to "cached chunks". Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
2015-02-18 17:15:50 +08:00
}
Integrate whole chunks into unused dirty page purging machinery. Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
2015-02-16 10:04:46 +08:00
#endif
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
Reference in New Issue Copy Permalink