project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4c95c953e2
server-skynet-source-3rd-je.../include/jemalloc/internal/jemalloc_internal_externs.h

77 lines
2.5 KiB
C
Raw Normal View History

Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
#ifndef JEMALLOC_INTERNAL_EXTERNS_H
#define JEMALLOC_INTERNAL_EXTERNS_H
Header refactoring: move atomic.h out of the catch-all
2017-04-11 10:04:40 +08:00
#include "jemalloc/internal/atomic.h"
Pull HPA options into a containing struct. Currently that just means max_alloc, but we're about to add more. While we're touching these lines anyways, tweak things to be more in line with testing.
2020-12-09 08:33:39 +08:00
#include "jemalloc/internal/hpa_opts.h"
SEC: Expand option configurability. This change pulls the SEC options into a struct, which simplifies their handling across various modules (e.g. PA needs to forward on SEC options from the malloc_conf string, but it doesn't really need to know their names). While we're here, make some of the fixed constants configurable, and unify naming from the configuration options to the internals.
2021-01-27 10:35:18 +08:00
#include "jemalloc/internal/sec_opts.h"
Header refactoring: tsd - cleanup and dependency breaking. This removes the tsd macros (which are used only for tsd_t in real builds). We break up the circular dependencies involving tsd. We also move all tsd access through getters and setters. This allows us to assert that we only touch data when tsd is in a valid state. We simplify the usages of the x macro trick, removing all the customizability (get/set, init, cleanup), moving the lifetime logic to tsd_init and tsd_cleanup. This lets us make initialization order independent of order within tsd_t.
2017-04-27 09:37:44 +08:00
#include "jemalloc/internal/tsd_types.h"
High Resolution Timestamps for Profiling
2020-06-02 21:42:44 +08:00
#include "jemalloc/internal/nstime.h"
Header refactoring: move atomic.h out of the catch-all
2017-04-11 10:04:40 +08:00
Bundle 3 branches on fast path into tsd_state. Added tsd_state_nominal_slow, which on fast path malloc() incorporates tcache_enabled check, and on fast path free() bundles both malloc_slow and tcache_enabled branches.
2017-04-12 14:13:45 +08:00
/* TSD checks this to set thread local slow state accordingly. */
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern bool malloc_slow;
Bundle 3 branches on fast path into tsd_state. Added tsd_state_nominal_slow, which on fast path malloc() incorporates tcache_enabled check, and on fast path free() bundles both malloc_slow and tcache_enabled branches.
2017-04-12 14:13:45 +08:00
/* Run-time options. */
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern bool opt_abort;
extern bool opt_abort_conf;
Add runtime detection for MADV_DONTNEED zeroes pages (mostly for qemu) qemu does not support this, yet [1], and you can get very tricky assert if you will run program with jemalloc in use under qemu: <jemalloc>: ../contrib/jemalloc/src/extent.c:1195: Failed assertion: "p[i] == 0" [1]: https://patchwork.kernel.org/patch/10576637/ Here is a simple example that shows the problem [2]: // Gist to check possible issues with MADV_DONTNEED // For example it does not supported by qemu user // There is a patch for this [1], but it hasn't been applied. // [1]: https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg05422.html #include <sys/mman.h> #include <stdio.h> #include <stddef.h> #include <assert.h> #include <string.h> int main(int argc, char **argv) { void *addr = mmap(NULL, 1<<16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return 1; } memset(addr, 'A', 1<<16); if (!madvise(addr, 1<<16, MADV_DONTNEED)) { puts("MADV_DONTNEED does not return error. Check memory."); for (int i = 0; i < 1<<16; ++i) { assert(((unsigned char *)addr)[i] == 0); } } else { perror("madvise"); } if (munmap(addr, 1<<16)) { perror("munmap"); return 1; } return 0; } ### unpatched qemu $ qemu-x86_64-static /tmp/test-MADV_DONTNEED MADV_DONTNEED does not return error. Check memory. test-MADV_DONTNEED: /tmp/test-MADV_DONTNEED.c:19: main: Assertion `((unsigned char *)addr)[i] == 0' failed. qemu: uncaught target signal 6 (Aborted) - core dumped Aborted (core dumped) ### patched qemu (by returning ENOSYS error) $ qemu-x86_64 /tmp/test-MADV_DONTNEED madvise: Success ### patch for qemu to return ENOSYS diff --git a/linux-user/syscall.c b/linux-user/syscall.c index 897d20c076..5540792e0e 100644 --- a/linux-user/syscall.c +++ b/linux-user/syscall.c @@ -11775,7 +11775,7 @@ static abi_long do_syscall1(void *cpu_env, int num, abi_long arg1, turns private file-backed mappings into anonymous mappings. This will break MADV_DONTNEED. This is a hint, so ignoring and returning success is ok. */ - return 0; + return ENOSYS; #endif #ifdef TARGET_NR_fcntl64 case TARGET_NR_fcntl64: [2]: https://gist.github.com/azat/12ba2c825b710653ece34dba7f926ece v2: - review fixes - add opt_dont_trust_madvise v3: - review fixes - rename opt_dont_trust_madvise to opt_trust_madvise
2020-12-19 03:23:35 +08:00
extern bool opt_trust_madvise;
Add confirm_conf option If the confirm_conf option is set, when the program starts, each of the four malloc_conf strings will be printed, and each option will be printed when being set.
2019-05-01 04:54:00 +08:00
extern bool opt_confirm_conf;
HPA: Tie components into a PAI implementation.
2020-08-15 04:36:41 +08:00
extern bool opt_hpa;
Pull HPA options into a containing struct. Currently that just means max_alloc, but we're about to add more. While we're touching these lines anyways, tweak things to be more in line with testing.
2020-12-09 08:33:39 +08:00
extern hpa_shard_opts_t opt_hpa_opts;
SEC: Expand option configurability. This change pulls the SEC options into a struct, which simplifies their handling across various modules (e.g. PA needs to forward on SEC options from the malloc_conf string, but it doesn't really need to know their names). While we're here, make some of the fixed constants configurable, and unify naming from the configuration options to the internals.
2021-01-27 10:35:18 +08:00
extern sec_opts_t opt_hpa_sec_opts;
PA: Use an SEC in fron of the HPA shard.
2020-10-17 04:14:59 +08:00
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern const char *opt_junk;
extern bool opt_junk_alloc;
extern bool opt_junk_free;
Move junking out of arena/tcache code. This is debug only and we keep it off the fast path. Moving it here simplifies the internal logic. This never tries to junk on regions that were shrunk via xallocx. I think this is fine for two reasons: - The shrunk-with-xallocx case is rare. - We don't always do that anyway before this diff (it depends on the opt settings and extent hooks in effect).
2020-02-29 03:37:39 +08:00
extern void (*junk_free_callback)(void *ptr, size_t size);
extern void (*junk_alloc_callback)(void *ptr, size_t size);
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern bool opt_utrace;
extern bool opt_xmalloc;
Add opt.experimental_infallible_new. This allows a guarantee that operator new never throws. Fix the .gitignore rules to include test/integration/cpp while we're here.
2021-06-22 04:40:30 +08:00
extern bool opt_experimental_infallible_new;
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern bool opt_zero;
extern unsigned opt_narenas;
Realloc: Make behavior of realloc(ptr, 0) configurable.
2019-09-24 08:56:19 +08:00
extern zero_realloc_action_t opt_zero_realloc_action;
Inline malloc fastpath into operator new. This saves a small but non-negligible amount of CPU in C++ programs.
2021-02-09 00:49:34 +08:00
extern malloc_init_t malloc_init_state;
Realloc: Make behavior of realloc(ptr, 0) configurable.
2019-09-24 08:56:19 +08:00
extern const char *zero_realloc_mode_names[];
Add stats counters for number of zero reallocs
2019-10-27 02:04:46 +08:00
extern atomic_zu_t zero_realloc_count;
Implement opt.cache_oblivious. Keep config.cache_oblivious for now to remain backward-compatible.
2021-02-10 14:24:35 +08:00
extern bool opt_cache_oblivious;
Add double free detection in thread cache for debug build Add new runtime option `debug_double_free_max_scan` that specifies the max number of stack entries to scan in the cache bit when trying to detect the double free bug (currently debug build only).
2022-07-21 06:25:56 +08:00
extern unsigned opt_debug_double_free_max_scan;
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
Implement use-after-free detection using junk and stash. On deallocation, sampled pointers (specially aligned) get junked and stashed into tcache (to prevent immediate reuse). The expected behavior is to have read-after-free corrupted and stopped by the junk-filling, while write-after-free is checked when flushing the stashed pointers.
2021-10-19 08:33:15 +08:00
/* Escape free-fastpath when ptr & mask == 0 (for sanitization purpose). */
extern uintptr_t san_cache_bin_nonfast_mask;
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
/* Number of CPUs. */
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern unsigned ncpus;
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
/* Number of arenas used for automatic multiplexing of threads and arenas. */
Added opt_abort_conf: abort on invalid config options.
2017-05-26 06:30:11 +08:00
extern unsigned narenas_auto;
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
Refactor arena_is_auto.
2018-06-02 06:06:36 +08:00
/* Base index for manual arenas. */
extern unsigned manual_arena_base;
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
/*
* Arenas that are used to service external requests. Not all elements of the
* arenas array are necessarily used; arenas are created lazily as needed.
*/
extern atomic_p_t arenas[];
void *a0malloc(size_t size);
void a0dalloc(void *ptr);
void *bootstrap_malloc(size_t size);
void *bootstrap_calloc(size_t num, size_t size);
void bootstrap_free(void *ptr);
void arena_set(unsigned ind, arena_t *arena);
unsigned narenas_total_get(void);
add experimental.arenas_create_ext mallctl This mallctl accepts an arena_config_t structure which can be used to customize the behavior of the arena. Right now it contains extent_hooks and a new option, metadata_use_hooks, which controls whether the extent hooks are also used for metadata allocation. The medata_use_hooks option has two main use cases: 1. In heterogeneous memory systems, to avoid metadata being placed on potentially slower memory. 2. Avoiding virtual memory from being leaked as a result of metadata allocation failure originating in an extent hook.
2021-08-23 20:03:35 +08:00
arena_t *arena_init(tsdn_t *tsdn, unsigned ind, const arena_config_t *config);
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
arena_t *arena_choose_hard(tsd_t *tsd, bool internal);
Simplify arena_migrate() to take arena_t* instead of indices. This makes debugging slightly easier and avoids the confusion of "should we create new arenas" here.
2022-01-11 05:34:07 +08:00
void arena_migrate(tsd_t *tsd, arena_t *oldarena, arena_t *newarena);
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
void iarena_cleanup(tsd_t *tsd);
void arena_cleanup(tsd_t *tsd);
Add batch allocation API
2020-04-24 06:46:45 +08:00
size_t batch_alloc(void **ptrs, size_t num, size_t size, int flags);
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
void jemalloc_prefork(void);
void jemalloc_postfork_parent(void);
void jemalloc_postfork_child(void);
remove compare and branch in fast path for c++ operator delete[] Summary: sdallocx is checking a flag that will never be set (at least in the provided C++ destructor implementation). This branch will probably only rarely be mispredicted however it removes two instructions in sdallocx and one at the callsite (to zero out flags).
2019-03-09 03:50:30 +08:00
void je_sdallocx_noflags(void *ptr, size_t size);
Inline malloc fastpath into operator new. This saves a small but non-negligible amount of CPU in C++ programs.
2021-02-09 00:49:34 +08:00
void *malloc_default(size_t size);
Header refactoring: Split up jemalloc_internal.h This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
2017-04-11 09:17:55 +08:00
#endif /* JEMALLOC_INTERNAL_EXTERNS_H */
Reference in New Issue Copy Permalink