project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
884fa22b8c
server-skynet-source-3rd-je.../src/large.c

349 lines
9.5 KiB
C
Raw Normal View History

Rename huge to large.
2016-06-01 05:50:21 +08:00
#define JEMALLOC_LARGE_C_
Restructure source tree.
2010-02-12 06:45:59 +08:00
#include "jemalloc/internal/jemalloc_internal.h"
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
/******************************************************************************/
void *
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_malloc(tsdn_t *tsdn, arena_t *arena, size_t usize, bool zero)
Add alignment support to chunk_alloc().
2012-04-11 01:50:33 +08:00
{
Make *allocx() size class overflow behavior defined. Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
2016-02-26 07:29:49 +08:00
assert(usize == s2u(usize));
Normalize size classes. Normalize size classes to use the same number of size classes per size doubling (currently hard coded to 4), across the intire range of size classes. Small size classes already used this spacing, but in order to support this change, additional small size classes now fill [4 KiB .. 16 KiB). Large size classes range from [16 KiB .. 4 MiB). Huge size classes now support non-multiples of the chunk size in order to fill (4 MiB .. 16 MiB).
2014-10-06 08:54:10 +08:00
Rename huge to large.
2016-06-01 05:50:21 +08:00
return (large_palloc(tsdn, arena, usize, CACHELINE, zero));
Add alignment support to chunk_alloc().
2012-04-11 01:50:33 +08:00
}
void *
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_palloc(tsdn_t *tsdn, arena_t *arena, size_t usize, size_t alignment,
Do not allocate metadata via non-auto arenas, nor tcaches. This assures that all internally allocated metadata come from the first opt_narenas arenas, i.e. the automatically multiplexed arenas.
2016-04-23 05:34:14 +08:00
bool zero)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
{
Make *allocx() size class overflow behavior defined. Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
2016-02-26 07:29:49 +08:00
size_t ausize;
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_t *extent;
Fix chunk allocation/deallocation bugs. Fix chunk_alloc_dss() to zero memory when requested. Fix chunk_dealloc() to avoid chunk_dealloc_mmap() for dss-allocated memory. Fix huge_palloc() to always junk fill when requested. Improve chunk_recycle() to report that memory is zeroed as a side effect of pages_purge().
2012-04-22 07:04:51 +08:00
bool is_zeroed;
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
UNUSED bool idump JEMALLOC_CC_SILENCE_INIT(false);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
assert(!tsdn_null(tsdn) || arena != NULL);
Make *allocx() size class overflow behavior defined. Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
2016-02-26 07:29:49 +08:00
ausize = sa2u(usize, alignment);
Rename huge to large.
2016-06-01 05:50:21 +08:00
if (unlikely(ausize == 0 || ausize > LARGE_MAXCLASS))
Fix huge_palloc() to handle size rather than usize input. huge_ralloc() passes a size that may not be precisely a size class, so make huge_palloc() handle the more general case of a size input rather than usize. This regression appears to have been introduced by the addition of in-place huge reallocation; as such it was never incorporated into a release.
2015-07-24 08:16:32 +08:00
return (NULL);
Fix chunk allocation/deallocation bugs. Fix chunk_alloc_dss() to zero memory when requested. Fix chunk_dealloc() to avoid chunk_dealloc_mmap() for dss-allocated memory. Fix huge_palloc() to always junk fill when requested. Improve chunk_recycle() to report that memory is zeroed as a side effect of pages_purge().
2012-04-22 07:04:51 +08:00
/*
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
* Copy zero into is_zeroed and pass the copy to extent_alloc(), so that
Fix chunk allocation/deallocation bugs. Fix chunk_alloc_dss() to zero memory when requested. Fix chunk_dealloc() to avoid chunk_dealloc_mmap() for dss-allocated memory. Fix huge_palloc() to always junk fill when requested. Improve chunk_recycle() to report that memory is zeroed as a side effect of pages_purge().
2012-04-22 07:04:51 +08:00
* it is possible to make correct junk/zero fill decisions below.
*/
is_zeroed = zero;
Guard tsdn_tsd() call with tsdn_null() check.
2016-05-12 07:52:58 +08:00
if (likely(!tsdn_null(tsdn)))
arena = arena_choose(tsdn_tsd(tsdn), arena);
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (unlikely(arena == NULL) || (extent = arena_extent_alloc_large(tsdn,
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
arena, usize, alignment, &is_zeroed)) == NULL)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
return (NULL);
Rename huge to large.
2016-06-01 05:50:21 +08:00
/* Insert extent into large. */
malloc_mutex_lock(tsdn, &arena->large_mtx);
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
ql_elm_new(extent, ql_link);
Rename huge to large.
2016-06-01 05:50:21 +08:00
ql_tail_insert(&arena->large, extent, ql_link);
malloc_mutex_unlock(tsdn, &arena->large_mtx);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
if (config_prof && arena_prof_accum(tsdn, arena, usize))
prof_idump(tsdn);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Purge/zero sub-chunk huge allocations as necessary. Purge trailing pages during shrinking huge reallocation when resulting size is not a multiple of the chunk size. Similarly, zero pages if necessary during growing huge reallocation when the resulting size is not a multiple of the chunk size.
2014-10-16 09:02:02 +08:00
if (zero || (config_fill && unlikely(opt_zero))) {
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
if (!is_zeroed) {
memset(extent_addr_get(extent), 0,
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
extent_usize_get(extent));
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
}
} else if (config_fill && unlikely(opt_junk_alloc)) {
memset(extent_addr_get(extent), JEMALLOC_ALLOC_JUNK,
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
extent_usize_get(extent));
Refactor chunk_dalloc_{cache,wrapper}() to take extent arguments. Rename arena_extent_[d]alloc() to extent_[d]alloc(). Move all chunk [de]registration responsibility into chunk.c.
2016-05-24 05:56:35 +08:00
}
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Resolve bootstrapping issues when embedded in FreeBSD libc. b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
2016-05-11 13:21:10 +08:00
arena_decay_tick(tsdn, arena);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
return (extent_addr_get(extent));
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
}
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
#ifdef JEMALLOC_JET
Rename huge to large.
2016-06-01 05:50:21 +08:00
#undef large_dalloc_junk
#define large_dalloc_junk JEMALLOC_N(n_large_dalloc_junk)
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
#endif
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
void
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc_junk(void *ptr, size_t usize)
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
{
memset(ptr, JEMALLOC_FREE_JUNK, usize);
}
#ifdef JEMALLOC_JET
Rename huge to large.
2016-06-01 05:50:21 +08:00
#undef large_dalloc_junk
#define large_dalloc_junk JEMALLOC_N(large_dalloc_junk)
large_dalloc_junk_t *large_dalloc_junk = JEMALLOC_N(n_large_dalloc_junk);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
#endif
Remove a stray memset(), and fix a junk filling test regression.
2016-06-04 11:04:30 +08:00
#ifdef JEMALLOC_JET
#undef large_dalloc_maybe_junk
#define large_dalloc_maybe_junk JEMALLOC_N(n_large_dalloc_maybe_junk)
#endif
void
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
large_dalloc_maybe_junk(void *ptr, size_t usize)
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
{
Introduce two new modes of junk filling: "alloc" and "free". In addition to true/false, opt.junk can now be either "alloc" or "free", giving applications the possibility of junking memory only on allocation or deallocation. This resolves #172.
2014-12-09 05:12:41 +08:00
if (config_fill && have_dss && unlikely(opt_junk_free)) {
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
/*
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
* Only bother junk filling if the extent isn't about to be
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
* unmapped.
*/
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
if (!config_munmap || (have_dss && extent_in_dss(ptr)))
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc_junk(ptr, usize);
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
}
}
Remove a stray memset(), and fix a junk filling test regression.
2016-06-04 11:04:30 +08:00
#ifdef JEMALLOC_JET
#undef large_dalloc_maybe_junk
#define large_dalloc_maybe_junk JEMALLOC_N(large_dalloc_maybe_junk)
large_dalloc_maybe_junk_t *large_dalloc_maybe_junk =
JEMALLOC_N(n_large_dalloc_maybe_junk);
#endif
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
static bool
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_ralloc_no_move_shrink(tsdn_t *tsdn, extent_t *extent, size_t usize)
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
{
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
arena_t *arena = extent_arena_get(extent);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
size_t oldusize = extent_usize_get(extent);
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t *extent_hooks = extent_hooks_get(arena);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
size_t diff = extent_size_get(extent) - (usize + large_pad);
Implement dynamic per arena control over dirty page purging. Add mallctls: - arenas.lg_dirty_mult is initialized via opt.lg_dirty_mult, and can be modified to change the initial lg_dirty_mult setting for newly created arenas. - arena.<i>.lg_dirty_mult controls an individual arena's dirty page purging threshold, and synchronously triggers any purging that may be necessary to maintain the constraint. - arena.<i>.chunk.purge allows the per arena dirty page purging function to be replaced. This resolves #93.
2015-03-19 09:55:33 +08:00
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
assert(oldusize > usize);
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
/* Split excess pages. */
if (diff != 0) {
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
extent_t *trail = extent_split_wrapper(tsdn, arena,
s/chunk_hook/extent_hook/g
2016-06-01 06:03:51 +08:00
&extent_hooks, extent, usize + large_pad, usize, diff,
diff);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
if (trail == NULL)
return (true);
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
Fix in-place shrinking huge reallocation purging bugs. Fix the shrinking case of huge_ralloc_no_move_similar() to purge the correct number of pages, at the correct offset. This regression was introduced by 8d6a3e8321a7767cb2ca0930b85d5d488a8cc659 (Implement dynamic per arena control over dirty page purging.). Fix huge_ralloc_no_move_shrink() to purge the correct number of pages. This bug was introduced by 9673983443a0782d975fbcb5d8457cfd411b8b56 (Purge/zero sub-chunk huge allocations as necessary.).
2015-03-26 09:56:55 +08:00
if (config_fill && unlikely(opt_junk_free)) {
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
large_dalloc_maybe_junk(extent_addr_get(trail),
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
extent_usize_get(trail));
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
arena_extent_cache_dalloc(tsdn, arena, &extent_hooks, trail);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
arena_extent_ralloc_large_shrink(tsdn, arena, extent, oldusize);
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
return (false);
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
}
Attempt to expand huge allocations in-place. This adds support for expanding huge allocations in-place by requesting memory at a specific address from the chunk allocator. It's currently only implemented for the chunk recycling path, although in theory it could also be done by optimistically allocating new chunks. On Linux, it could attempt an in-place mremap. However, that won't work in practice since the heap is grown downwards and memory is not unmapped (in a normal build, at least). Repeated vector reallocation micro-benchmark: #include <string.h> #include <stdlib.h> int main(void) { for (size_t i = 0; i < 100; i++) { void *ptr = NULL; size_t old_size = 0; for (size_t size = 4; size < (1 << 30); size *= 2) { ptr = realloc(ptr, size); if (!ptr) return 1; memset(ptr + old_size, 0xff, size - old_size); old_size = size; } free(ptr); } } The glibc allocator fails to do any in-place reallocations on this benchmark once it passes the M_MMAP_THRESHOLD (default 128k) but it elides the cost of copies via mremap, which is currently not something that jemalloc can use. With this improvement, jemalloc still fails to do any in-place huge reallocations for the first outer loop, but then succeeds 100% of the time for the remaining 99 iterations. The time spent doing allocations and copies drops down to under 5%, with nearly all of it spent doing purging + faulting (when huge pages are disabled) and the array memset. An improved mremap API (MREMAP_RETAIN - #138) would be far more general but this is a portable optimization and would still be useful on Linux for xallocx. Numbers with transparent huge pages enabled: glibc (copies elided via MREMAP_MAYMOVE): 8.471s jemalloc: 17.816s jemalloc + no-op madvise: 13.236s jemalloc + this commit: 6.787s jemalloc + this commit + no-op madvise: 6.144s Numbers with transparent huge pages disabled: glibc (copies elided via MREMAP_MAYMOVE): 15.403s jemalloc: 39.456s jemalloc + no-op madvise: 12.768s jemalloc + this commit: 15.534s jemalloc + this commit + no-op madvise: 6.354s Closes #137
2014-10-04 13:39:32 +08:00
static bool
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_ralloc_no_move_expand(tsdn_t *tsdn, extent_t *extent, size_t usize,
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
bool zero)
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
{
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
arena_t *arena = extent_arena_get(extent);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
size_t oldusize = extent_usize_get(extent);
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
bool is_zeroed_trail = false;
Modify extent hook functions to take an (extent_t *) argument. This facilitates the application accessing its own extent allocator metadata during hook invocations. This resolves #259.
2016-06-04 03:05:53 +08:00
extent_hooks_t *extent_hooks = extent_hooks_get(arena);
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
size_t trailsize = usize - extent_usize_get(extent);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
extent_t *trail;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if ((trail = arena_extent_cache_alloc(tsdn, arena, &extent_hooks,
Fix extent_alloc_cache[_locked]() to support decommitted allocation. Fix extent_alloc_cache[_locked]() to support decommitted allocation, and use this ability in arena_stash_dirty(), so that decommitted extents are not needlessly committed during purging. In practice this does not happen on any currently supported systems, because both extent merging and decommit must be implemented; all supported systems implement one xor the other.
2016-11-04 08:25:54 +08:00
extent_past_get(extent), trailsize, CACHELINE, &is_zeroed_trail)) ==
NULL) {
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
bool commit = true;
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if ((trail = extent_alloc_wrapper(tsdn, arena, &extent_hooks,
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
extent_past_get(extent), trailsize, 0, CACHELINE,
&is_zeroed_trail, &commit, false)) == NULL)
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
return (true);
}
Attempt to expand huge allocations in-place. This adds support for expanding huge allocations in-place by requesting memory at a specific address from the chunk allocator. It's currently only implemented for the chunk recycling path, although in theory it could also be done by optimistically allocating new chunks. On Linux, it could attempt an in-place mremap. However, that won't work in practice since the heap is grown downwards and memory is not unmapped (in a normal build, at least). Repeated vector reallocation micro-benchmark: #include <string.h> #include <stdlib.h> int main(void) { for (size_t i = 0; i < 100; i++) { void *ptr = NULL; size_t old_size = 0; for (size_t size = 4; size < (1 << 30); size *= 2) { ptr = realloc(ptr, size); if (!ptr) return 1; memset(ptr + old_size, 0xff, size - old_size); old_size = size; } free(ptr); } } The glibc allocator fails to do any in-place reallocations on this benchmark once it passes the M_MMAP_THRESHOLD (default 128k) but it elides the cost of copies via mremap, which is currently not something that jemalloc can use. With this improvement, jemalloc still fails to do any in-place huge reallocations for the first outer loop, but then succeeds 100% of the time for the remaining 99 iterations. The time spent doing allocations and copies drops down to under 5%, with nearly all of it spent doing purging + faulting (when huge pages are disabled) and the array memset. An improved mremap API (MREMAP_RETAIN - #138) would be far more general but this is a portable optimization and would still be useful on Linux for xallocx. Numbers with transparent huge pages enabled: glibc (copies elided via MREMAP_MAYMOVE): 8.471s jemalloc: 17.816s jemalloc + no-op madvise: 13.236s jemalloc + this commit: 6.787s jemalloc + this commit + no-op madvise: 6.144s Numbers with transparent huge pages disabled: glibc (copies elided via MREMAP_MAYMOVE): 15.403s jemalloc: 39.456s jemalloc + no-op madvise: 12.768s jemalloc + this commit: 15.534s jemalloc + this commit + no-op madvise: 6.354s Closes #137
2014-10-04 13:39:32 +08:00
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
if (extent_merge_wrapper(tsdn, arena, &extent_hooks, extent, trail)) {
extent_dalloc_wrapper(tsdn, arena, &extent_hooks, trail);
Fix huge allocation statistics.
2014-10-15 13:20:00 +08:00
return (true);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
}
Attempt to expand huge allocations in-place. This adds support for expanding huge allocations in-place by requesting memory at a specific address from the chunk allocator. It's currently only implemented for the chunk recycling path, although in theory it could also be done by optimistically allocating new chunks. On Linux, it could attempt an in-place mremap. However, that won't work in practice since the heap is grown downwards and memory is not unmapped (in a normal build, at least). Repeated vector reallocation micro-benchmark: #include <string.h> #include <stdlib.h> int main(void) { for (size_t i = 0; i < 100; i++) { void *ptr = NULL; size_t old_size = 0; for (size_t size = 4; size < (1 << 30); size *= 2) { ptr = realloc(ptr, size); if (!ptr) return 1; memset(ptr + old_size, 0xff, size - old_size); old_size = size; } free(ptr); } } The glibc allocator fails to do any in-place reallocations on this benchmark once it passes the M_MMAP_THRESHOLD (default 128k) but it elides the cost of copies via mremap, which is currently not something that jemalloc can use. With this improvement, jemalloc still fails to do any in-place huge reallocations for the first outer loop, but then succeeds 100% of the time for the remaining 99 iterations. The time spent doing allocations and copies drops down to under 5%, with nearly all of it spent doing purging + faulting (when huge pages are disabled) and the array memset. An improved mremap API (MREMAP_RETAIN - #138) would be far more general but this is a portable optimization and would still be useful on Linux for xallocx. Numbers with transparent huge pages enabled: glibc (copies elided via MREMAP_MAYMOVE): 8.471s jemalloc: 17.816s jemalloc + no-op madvise: 13.236s jemalloc + this commit: 6.787s jemalloc + this commit + no-op madvise: 6.144s Numbers with transparent huge pages disabled: glibc (copies elided via MREMAP_MAYMOVE): 15.403s jemalloc: 39.456s jemalloc + no-op madvise: 12.768s jemalloc + this commit: 15.534s jemalloc + this commit + no-op madvise: 6.354s Closes #137
2014-10-04 13:39:32 +08:00
Purge/zero sub-chunk huge allocations as necessary. Purge trailing pages during shrinking huge reallocation when resulting size is not a multiple of the chunk size. Similarly, zero pages if necessary during growing huge reallocation when the resulting size is not a multiple of the chunk size.
2014-10-16 09:02:02 +08:00
if (zero || (config_fill && unlikely(opt_zero))) {
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
if (config_cache_oblivious) {
/*
* Zero the trailing bytes of the original allocation's
* last page, since they are in an indeterminate state.
* There will always be trailing bytes, because ptr's
Move slabs out of chunks.
2016-05-30 09:34:50 +08:00
* offset from the beginning of the extent is a multiple
* of CACHELINE in [0 .. PAGE).
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
*/
void *zbase = (void *)
((uintptr_t)extent_addr_get(extent) + oldusize);
void *zpast = PAGE_ADDR2BASE((void *)((uintptr_t)zbase +
PAGE));
size_t nzero = (uintptr_t)zpast - (uintptr_t)zbase;
assert(nzero > 0);
memset(zbase, 0, nzero);
}
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
if (!is_zeroed_trail) {
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
memset((void *)((uintptr_t)extent_addr_get(extent) +
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
oldusize), 0, usize - oldusize);
Purge/zero sub-chunk huge allocations as necessary. Purge trailing pages during shrinking huge reallocation when resulting size is not a multiple of the chunk size. Similarly, zero pages if necessary during growing huge reallocation when the resulting size is not a multiple of the chunk size.
2014-10-16 09:02:02 +08:00
}
Introduce two new modes of junk filling: "alloc" and "free". In addition to true/false, opt.junk can now be either "alloc" or "free", giving applications the possibility of junking memory only on allocation or deallocation. This resolves #172.
2014-12-09 05:12:41 +08:00
} else if (config_fill && unlikely(opt_junk_alloc)) {
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
memset((void *)((uintptr_t)extent_addr_get(extent) + oldusize),
JEMALLOC_ALLOC_JUNK, usize - oldusize);
Fix pointer arithmetic undefined behavior. Reported by Denis Denisov.
2014-11-18 01:54:49 +08:00
}
Add per size class huge allocation statistics. Add per size class huge allocation statistics, and normalize various stats: - Change the arenas.nlruns type from size_t to unsigned. - Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's. - Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with stats.arenas.<i>.bins.<j>.curregs . - Add the stats.arenas.<i>.hchunks.<j>.nmalloc, stats.arenas.<i>.hchunks.<j>.ndalloc, stats.arenas.<i>.hchunks.<j>.nrequests, and stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
2014-10-13 13:53:59 +08:00
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
arena_extent_ralloc_large_expand(tsdn, arena, extent, oldusize);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
Attempt to expand huge allocations in-place. This adds support for expanding huge allocations in-place by requesting memory at a specific address from the chunk allocator. It's currently only implemented for the chunk recycling path, although in theory it could also be done by optimistically allocating new chunks. On Linux, it could attempt an in-place mremap. However, that won't work in practice since the heap is grown downwards and memory is not unmapped (in a normal build, at least). Repeated vector reallocation micro-benchmark: #include <string.h> #include <stdlib.h> int main(void) { for (size_t i = 0; i < 100; i++) { void *ptr = NULL; size_t old_size = 0; for (size_t size = 4; size < (1 << 30); size *= 2) { ptr = realloc(ptr, size); if (!ptr) return 1; memset(ptr + old_size, 0xff, size - old_size); old_size = size; } free(ptr); } } The glibc allocator fails to do any in-place reallocations on this benchmark once it passes the M_MMAP_THRESHOLD (default 128k) but it elides the cost of copies via mremap, which is currently not something that jemalloc can use. With this improvement, jemalloc still fails to do any in-place huge reallocations for the first outer loop, but then succeeds 100% of the time for the remaining 99 iterations. The time spent doing allocations and copies drops down to under 5%, with nearly all of it spent doing purging + faulting (when huge pages are disabled) and the array memset. An improved mremap API (MREMAP_RETAIN - #138) would be far more general but this is a portable optimization and would still be useful on Linux for xallocx. Numbers with transparent huge pages enabled: glibc (copies elided via MREMAP_MAYMOVE): 8.471s jemalloc: 17.816s jemalloc + no-op madvise: 13.236s jemalloc + this commit: 6.787s jemalloc + this commit + no-op madvise: 6.144s Numbers with transparent huge pages disabled: glibc (copies elided via MREMAP_MAYMOVE): 15.403s jemalloc: 39.456s jemalloc + no-op madvise: 12.768s jemalloc + this commit: 15.534s jemalloc + this commit + no-op madvise: 6.354s Closes #137
2014-10-04 13:39:32 +08:00
return (false);
}
Extract profiling code from [re]allocation functions. Extract profiling code from malloc(), imemalign(), calloc(), realloc(), mallocx(), rallocx(), and xallocx(). This slightly reduces the amount of code compiled into the fast paths, but the primary benefit is the combinatorial complexity reduction. Simplify iralloc[t]() by creating a separate ixalloc() that handles the no-move cases. Further simplify [mrxn]allocx() (and by implication [mrn]allocm()) to make request size overflows due to size class and/or alignment constraints trigger undefined behavior (detected by debug-only assertions). Report ENOMEM rather than EINVAL if an OOM occurs during heap profiling backtrace creation in imemalign(). This bug impacted posix_memalign() and aligned_alloc().
2014-01-13 07:05:44 +08:00
bool
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_ralloc_no_move(tsdn_t *tsdn, extent_t *extent, size_t usize_min,
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
size_t usize_max, bool zero)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
{
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
assert(s2u(extent_usize_get(extent)) == extent_usize_get(extent));
Make *allocx() size class overflow behavior defined. Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
2016-02-26 07:29:49 +08:00
/* The following should have been caught by callers. */
Rename huge to large.
2016-06-01 05:50:21 +08:00
assert(usize_min > 0 && usize_max <= LARGE_MAXCLASS);
/* Both allocation sizes must be large to avoid a move. */
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
assert(extent_usize_get(extent) >= LARGE_MINCLASS && usize_max >=
LARGE_MINCLASS);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
if (usize_max > extent_usize_get(extent)) {
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
/* Attempt to expand the allocation in-place. */
Rename huge to large.
2016-06-01 05:50:21 +08:00
if (!large_ralloc_no_move_expand(tsdn, extent, usize_max,
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
zero)) {
Use rtree-based chunk lookups rather than pointer bit twiddling. Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
2016-03-24 11:29:33 +08:00
arena_decay_tick(tsdn, extent_arena_get(extent));
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
return (false);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
}
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
/* Try again, this time with usize_min. */
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
if (usize_min < usize_max && usize_min >
Rename huge to large.
2016-06-01 05:50:21 +08:00
extent_usize_get(extent) &&
large_ralloc_no_move_expand(tsdn, extent, usize_min,
zero)) {
Use rtree-based chunk lookups rather than pointer bit twiddling. Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
2016-03-24 11:29:33 +08:00
arena_decay_tick(tsdn, extent_arena_get(extent));
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
return (false);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
}
Normalize size classes. Normalize size classes to use the same number of size classes per size doubling (currently hard coded to 4), across the intire range of size classes. Small size classes already used this spacing, but in order to support this change, additional small size classes now fill [4 KiB .. 16 KiB). Large size classes range from [16 KiB .. 4 MiB). Huge size classes now support non-multiples of the chunk size in order to fill (4 MiB .. 16 MiB).
2014-10-06 08:54:10 +08:00
}
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
/*
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
* Avoid moving the allocation if the existing extent size accommodates
Normalize size classes. Normalize size classes to use the same number of size classes per size doubling (currently hard coded to 4), across the intire range of size classes. Small size classes already used this spacing, but in order to support this change, additional small size classes now fill [4 KiB .. 16 KiB). Large size classes range from [16 KiB .. 4 MiB). Huge size classes now support non-multiples of the chunk size in order to fill (4 MiB .. 16 MiB).
2014-10-06 08:54:10 +08:00
* the new size.
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
*/
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
if (extent_usize_get(extent) >= usize_min && extent_usize_get(extent) <=
Allow chunks to not be naturally aligned. Precisely size extents for huge size classes that aren't multiples of chunksize.
2016-05-27 13:12:38 +08:00
usize_max) {
Use rtree-based chunk lookups rather than pointer bit twiddling. Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
2016-03-24 11:29:33 +08:00
arena_decay_tick(tsdn, extent_arena_get(extent));
Extract profiling code from [re]allocation functions. Extract profiling code from malloc(), imemalign(), calloc(), realloc(), mallocx(), rallocx(), and xallocx(). This slightly reduces the amount of code compiled into the fast paths, but the primary benefit is the combinatorial complexity reduction. Simplify iralloc[t]() by creating a separate ixalloc() that handles the no-move cases. Further simplify [mrxn]allocx() (and by implication [mrn]allocm()) to make request size overflows due to size class and/or alignment constraints trigger undefined behavior (detected by debug-only assertions). Report ENOMEM rather than EINVAL if an OOM occurs during heap profiling backtrace creation in imemalign(). This bug impacted posix_memalign() and aligned_alloc().
2014-01-13 07:05:44 +08:00
return (false);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
}
Generalize chunk management hooks. Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
2015-07-28 23:28:19 +08:00
/* Attempt to shrink the allocation in-place. */
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
if (extent_usize_get(extent) > usize_max) {
Rename huge to large.
2016-06-01 05:50:21 +08:00
if (!large_ralloc_no_move_shrink(tsdn, extent, usize_max)) {
Use rtree-based chunk lookups rather than pointer bit twiddling. Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
2016-03-24 11:29:33 +08:00
arena_decay_tick(tsdn, extent_arena_get(extent));
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
return (false);
}
}
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
return (true);
}
Implement in-place huge allocation shrinking. Trivial example: #include <stdlib.h> int main(void) { void *ptr = malloc(1024 * 1024 * 8); if (!ptr) return 1; ptr = realloc(ptr, 1024 * 1024 * 4); if (!ptr) return 1; } Before: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000 madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0 After: mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000 madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0 Closes #134
2014-09-30 22:33:46 +08:00
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
static void *
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_ralloc_move_helper(tsdn_t *tsdn, arena_t *arena, size_t usize,
Do not allocate metadata via non-auto arenas, nor tcaches. This assures that all internally allocated metadata come from the first opt_narenas arenas, i.e. the automatically multiplexed arenas.
2016-04-23 05:34:14 +08:00
size_t alignment, bool zero)
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
{
Attempt to expand huge allocations in-place. This adds support for expanding huge allocations in-place by requesting memory at a specific address from the chunk allocator. It's currently only implemented for the chunk recycling path, although in theory it could also be done by optimistically allocating new chunks. On Linux, it could attempt an in-place mremap. However, that won't work in practice since the heap is grown downwards and memory is not unmapped (in a normal build, at least). Repeated vector reallocation micro-benchmark: #include <string.h> #include <stdlib.h> int main(void) { for (size_t i = 0; i < 100; i++) { void *ptr = NULL; size_t old_size = 0; for (size_t size = 4; size < (1 << 30); size *= 2) { ptr = realloc(ptr, size); if (!ptr) return 1; memset(ptr + old_size, 0xff, size - old_size); old_size = size; } free(ptr); } } The glibc allocator fails to do any in-place reallocations on this benchmark once it passes the M_MMAP_THRESHOLD (default 128k) but it elides the cost of copies via mremap, which is currently not something that jemalloc can use. With this improvement, jemalloc still fails to do any in-place huge reallocations for the first outer loop, but then succeeds 100% of the time for the remaining 99 iterations. The time spent doing allocations and copies drops down to under 5%, with nearly all of it spent doing purging + faulting (when huge pages are disabled) and the array memset. An improved mremap API (MREMAP_RETAIN - #138) would be far more general but this is a portable optimization and would still be useful on Linux for xallocx. Numbers with transparent huge pages enabled: glibc (copies elided via MREMAP_MAYMOVE): 8.471s jemalloc: 17.816s jemalloc + no-op madvise: 13.236s jemalloc + this commit: 6.787s jemalloc + this commit + no-op madvise: 6.144s Numbers with transparent huge pages disabled: glibc (copies elided via MREMAP_MAYMOVE): 15.403s jemalloc: 39.456s jemalloc + no-op madvise: 12.768s jemalloc + this commit: 15.534s jemalloc + this commit + no-op madvise: 6.354s Closes #137
2014-10-04 13:39:32 +08:00
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
if (alignment <= CACHELINE)
Rename huge to large.
2016-06-01 05:50:21 +08:00
return (large_malloc(tsdn, arena, usize, zero));
return (large_palloc(tsdn, arena, usize, alignment, zero));
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
}
void *
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_ralloc(tsdn_t *tsdn, arena_t *arena, extent_t *extent, size_t usize,
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
size_t alignment, bool zero, tcache_t *tcache)
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
{
void *ret;
size_t copysize;
Make *allocx() size class overflow behavior defined. Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
2016-02-26 07:29:49 +08:00
/* The following should have been caught by callers. */
Rename huge to large.
2016-06-01 05:50:21 +08:00
assert(usize > 0 && usize <= LARGE_MAXCLASS);
/* Both allocation sizes must be large to avoid a move. */
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
assert(extent_usize_get(extent) >= LARGE_MINCLASS && usize >=
LARGE_MINCLASS);
Make *allocx() size class overflow behavior defined. Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
2016-02-26 07:29:49 +08:00
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
/* Try to avoid moving the allocation. */
Rename huge to large.
2016-06-01 05:50:21 +08:00
if (!large_ralloc_no_move(tsdn, extent, usize, usize, zero))
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
return (extent_addr_get(extent));
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
/*
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
* usize and old size are different enough that we need to use a
Add {,r,s,d}allocm(). Add allocm(), rallocm(), sallocm(), and dallocm(), which are a functional superset of malloc(), calloc(), posix_memalign(), malloc_usable_size(), and free().
2010-09-18 06:46:18 +08:00
* different size class. In that case, fall back to allocating new
* space and copying.
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
*/
Rename huge to large.
2016-06-01 05:50:21 +08:00
ret = large_ralloc_move_helper(tsdn, arena, usize, alignment, zero);
Fix xallocx() bugs. Fix xallocx() bugs related to the 'extra' parameter when specified as non-zero.
2015-09-12 07:18:53 +08:00
if (ret == NULL)
return (NULL);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
copysize = (usize < extent_usize_get(extent)) ? usize :
extent_usize_get(extent);
Add/use chunk_split_wrapper(). Remove redundant ptr/oldsize args from huge_*(). Refactor huge/chunk/arena code boundaries.
2016-05-19 12:02:46 +08:00
memcpy(ret, extent_addr_get(extent), copysize);
isdalloct(tsdn, extent, extent_addr_get(extent),
Implement cache-oblivious support for huge size classes.
2016-05-28 09:57:15 +08:00
extent_usize_get(extent), tcache, true);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
return (ret);
}
Fix large_dalloc_impl() to always lock large_mtx.
2016-09-24 03:16:55 +08:00
/*
* junked_locked indicates whether the extent's data have been junk-filled, and
* whether the arena's lock is currently held. The arena's large_mtx is
* independent of these considerations.
*/
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
static void
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc_impl(tsdn_t *tsdn, extent_t *extent, bool junked_locked)
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
{
Move centralized chunk management into arenas. Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
2015-02-12 04:24:27 +08:00
arena_t *arena;
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
arena = extent_arena_get(extent);
Fix large_dalloc_impl() to always lock large_mtx.
2016-09-24 03:16:55 +08:00
malloc_mutex_lock(tsdn, &arena->large_mtx);
Rename huge to large.
2016-06-01 05:50:21 +08:00
ql_remove(&arena->large, extent, ql_link);
Fix large_dalloc_impl() to always lock large_mtx.
2016-09-24 03:16:55 +08:00
malloc_mutex_unlock(tsdn, &arena->large_mtx);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
if (!junked_locked) {
Make dss operations lockless. Rather than protecting dss operations with a mutex, use atomic operations. This has negligible impact on synchronization overhead during typical dss allocation, but is a substantial improvement for extent_in_dss() and the newly added extent_dss_mergeable(), which can be called multiple times during extent deallocations. This change also has the advantage of avoiding tsd in deallocation paths associated with purging, which resolves potential deadlocks during thread exit due to attempted tsd resurrection. This resolves #425.
2016-10-14 03:18:38 +08:00
large_dalloc_maybe_junk(extent_addr_get(extent),
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
extent_usize_get(extent));
}
Rename most remaining *chunk* APIs to *extent*.
2016-06-02 03:59:02 +08:00
arena_extent_dalloc_large(tsdn, arena, extent, junked_locked);
Implement decay-based unused dirty page purging. This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
2016-02-20 12:09:31 +08:00
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
if (!junked_locked)
arena_decay_tick(tsdn, arena);
}
void
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc_junked_locked(tsdn_t *tsdn, extent_t *extent)
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
{
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc_impl(tsdn, extent, true);
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
}
void
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc(tsdn_t *tsdn, extent_t *extent)
Use huge size class infrastructure for large size classes.
2016-05-28 15:17:28 +08:00
{
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_dalloc_impl(tsdn, extent, false);
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
}
Implement metadata statistics. There are three categories of metadata: - Base allocations are used for bootstrap-sensitive internal allocator data structures. - Arena chunk headers comprise pages which track the states of the non-metadata pages. - Internal allocations differ from application-originated allocations in that they are for internal use, and that they are omitted from heap profiles. The metadata statistics comprise the metadata categories as follows: - stats.metadata: All metadata -- base + arena chunk headers + internal allocations. - stats.arenas.<i>.metadata.mapped: Arena chunk headers. - stats.arenas.<i>.metadata.allocated: Internal allocations. This is reported separately from the other metadata statistics because it overlaps with the allocated and active statistics, whereas the other metadata statistics do not. Base allocations are not reported separately, though their magnitude can be computed by subtracting the arena-specific metadata. This resolves #163.
2014-11-28 03:22:36 +08:00
size_t
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_salloc(tsdn_t *tsdn, const extent_t *extent)
Implement metadata statistics. There are three categories of metadata: - Base allocations are used for bootstrap-sensitive internal allocator data structures. - Arena chunk headers comprise pages which track the states of the non-metadata pages. - Internal allocations differ from application-originated allocations in that they are for internal use, and that they are omitted from heap profiles. The metadata statistics comprise the metadata categories as follows: - stats.metadata: All metadata -- base + arena chunk headers + internal allocations. - stats.arenas.<i>.metadata.mapped: Arena chunk headers. - stats.arenas.<i>.metadata.allocated: Internal allocations. This is reported separately from the other metadata statistics because it overlaps with the allocated and active statistics, whereas the other metadata statistics do not. Base allocations are not reported separately, though their magnitude can be computed by subtracting the arena-specific metadata. This resolves #163.
2014-11-28 03:22:36 +08:00
{
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
Remove some unnecessary locking.
2016-06-02 07:24:49 +08:00
return (extent_usize_get(extent));
Refactor jemalloc.c into multiple source files. Fix a stats bug in large object curruns accounting. Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM error path. Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-17 01:53:50 +08:00
}
Implement per thread heap profiling. Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
2014-08-19 07:22:13 +08:00
prof_tctx_t *
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_prof_tctx_get(tsdn_t *tsdn, const extent_t *extent)
Implement allocation profiling and leack checking. Add the --enable-prof and --enable-prof-libunwind configure options. Add the B/b, F/f, I/i, L/l, and U/u JEMALLOC_OPTIONS. Interval-based profile dump triggering is not yet implemented. Add supporting generic code: * Add memory barriers. * Add prn (LCG PRNG). * Add hash (Murmur hash function). * Add ckh (cuckoo hash tables).
2010-02-11 02:37:56 +08:00
{
Fix fork-related bugs. Acquire/release arena bin locks as part of the prefork/postfork. This bug made deadlock in the child between fork and exec a possibility. Split jemalloc_postfork() into jemalloc_postfork_{parent,child}() so that the child can reinitialize mutexes rather than unlocking them. In practice, this bug tended not to cause problems.
2012-03-14 07:31:41 +08:00
Remove some unnecessary locking.
2016-06-02 07:24:49 +08:00
return (extent_prof_tctx_get(extent));
Fix fork-related bugs. Acquire/release arena bin locks as part of the prefork/postfork. This bug made deadlock in the child between fork and exec a possibility. Split jemalloc_postfork() into jemalloc_postfork_{parent,child}() so that the child can reinitialize mutexes rather than unlocking them. In practice, this bug tended not to cause problems.
2012-03-14 07:31:41 +08:00
}
void
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_prof_tctx_set(tsdn_t *tsdn, extent_t *extent, prof_tctx_t *tctx)
Fix fork-related bugs. Acquire/release arena bin locks as part of the prefork/postfork. This bug made deadlock in the child between fork and exec a possibility. Split jemalloc_postfork() into jemalloc_postfork_{parent,child}() so that the child can reinitialize mutexes rather than unlocking them. In practice, this bug tended not to cause problems.
2012-03-14 07:31:41 +08:00
{
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_prof_tctx_set(extent, tctx);
Fix fork-related bugs. Acquire/release arena bin locks as part of the prefork/postfork. This bug made deadlock in the child between fork and exec a possibility. Split jemalloc_postfork() into jemalloc_postfork_{parent,child}() so that the child can reinitialize mutexes rather than unlocking them. In practice, this bug tended not to cause problems.
2012-03-14 07:31:41 +08:00
}
Resolve an unsupported special case in arena_prof_tctx_set(). Add arena_prof_tctx_reset() and use it instead of arena_prof_tctx_set() when resetting the tctx pointer during reallocation, which happens whenever an originally sampled reallocated object is not sampled during reallocation. This regression was introduced by 594c759f37c301d0245dc2accf4d4aaf9d202819 (Optimize arena_prof_tctx_set().)
2015-09-15 14:48:11 +08:00
void
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_prof_tctx_reset(tsdn_t *tsdn, extent_t *extent)
Resolve an unsupported special case in arena_prof_tctx_set(). Add arena_prof_tctx_reset() and use it instead of arena_prof_tctx_set() when resetting the tctx pointer during reallocation, which happens whenever an originally sampled reallocated object is not sampled during reallocation. This regression was introduced by 594c759f37c301d0245dc2accf4d4aaf9d202819 (Optimize arena_prof_tctx_set().)
2015-09-15 14:48:11 +08:00
{
Rename huge to large.
2016-06-01 05:50:21 +08:00
large_prof_tctx_set(tsdn, extent, (prof_tctx_t *)(uintptr_t)1U);
Resolve an unsupported special case in arena_prof_tctx_set(). Add arena_prof_tctx_reset() and use it instead of arena_prof_tctx_set() when resetting the tctx pointer during reallocation, which happens whenever an originally sampled reallocated object is not sampled during reallocation. This regression was introduced by 594c759f37c301d0245dc2accf4d4aaf9d202819 (Optimize arena_prof_tctx_set().)
2015-09-15 14:48:11 +08:00
}
Reference in New Issue Copy Permalink