David Goldblatt
259c5e3e8f
psset: Add stats
2020-09-18 12:39:25 -07:00
David Goldblatt
018b162d67
Add psset: a set of pageslabs.
...
This introduces a new sort of edata_t; a pageslab, and a set to manage them.
This is part of a series of a commits to implement a hugepage allocator; the
pageset will be per-arena, and track small page allocations requests within a
larger extent allocated from a centralized hugepage allocator.
2020-09-18 12:39:25 -07:00
David Goldblatt
ed99d300b9
Flat bitmap: Add longest-range computation.
...
This will come in handy in the (upcoming) page-slab set assertions.
2020-09-18 12:39:25 -07:00
David Goldblatt
e034500698
Edata: rename "ranged" bit to "pai".
...
This better represents its intended purpose; the hugepage allocator design
evolved away from needing contiguity of hugepage virtual address space.
2020-09-18 12:39:25 -07:00
David Goldblatt
7ad2f78663
Avoid a -Wundef warning on LG_SLAB_MAXREGS.
2020-09-17 10:05:40 -07:00
Hao Liu
1541ffc765
configure: add --with-lg-slab-maxregs configure option.
...
Specify the maximum number of regions in a slab, which is
(<lg-page> - <lg-tiny-min>) by default. This increases the limit of slab sizes
specified by "slab_sizes" in malloc_conf. This should never be less than
the default value. The max value of this option is related to LG_BITMAP_MAXBITS
(see more in bitmap.h).
For example, on a 4k page size system, if we:
1) configure jemalloc with with --with-lg-slab-maxregs=12.
2) export MALLOC_CONF="slab_sizes:9-16:4"
The slab size of 16 bytes is set to 4 pages. Previously, the default
lg-slab-maxregs is 9 (i.e. 12 - 3). The max slab size of 16 bytes is 2 pages
(i.e. (1<<9) * 16 bytes). By increasing the value from 9 to 12, the max slab
size can be set by MALLOC_CONF is 16 pages (i.e. (1<<12) * 16 bytes).
2020-09-16 13:58:38 -07:00
Yinan Zhang
b549389e4a
Correct usize in prof last-N record
2020-09-09 13:31:35 -07:00
Yinan Zhang
866231fc61
Do not repeat reentrancy test in profiling
2020-08-25 16:49:32 -07:00
Yinan Zhang
20f2479ed7
Do not create size class tables for non-prof builds
2020-08-24 20:10:02 -07:00
Yinan Zhang
8efcdc3f98
Move unbias data to prof_data
2020-08-24 20:10:02 -07:00
David Goldblatt
5e90fd006e
Geom_grow: Don't keep the mutex internal.
...
We're about to use it in ways that will have external synchronization.
2020-08-19 16:53:21 -07:00
David Goldblatt
c57494879f
Geom_grow: Don't take tsdn at init.
...
It's never used.
2020-08-19 16:53:21 -07:00
David Goldblatt
ffe552223c
Geom_grow: Move in advancing logic.
2020-08-19 16:53:21 -07:00
David Goldblatt
131b1b5338
Rename ecache_grow -> geom_grow.
...
We're about to start using it outside of the ecaches, in the HPA central
allocator.
2020-08-19 16:53:21 -07:00
David Goldblatt
9e18ae639f
Config: safety checks don't imply size checks.
...
The commit introducing size checks accidentally enabled them whenever any safety
checks were on. This ends up causing the regression that splitting up the
features was intended to avoid. Fix the issue.
2020-08-12 13:00:19 -07:00
David Goldblatt
eaed1e39be
Add sized-delete size-checking functionality.
...
The existing checks are good at finding such issues (on tcache flush), but not
so good at pinpointing them. Debug mode can find them, but sometimes debug mode
slows down a program so much that hard-to-hit bugs can take a long time to
crash.
This commit adds functionality to keep programs mostly on their fast paths,
while also checking every sized delete argument they get.
2020-08-05 19:34:05 -07:00
David Goldblatt
60993697d8
Prof: Add prof_unbias.
...
This gives more accurate attribution of bytes and counts to stack traces,
without introducing backwards incompatibilities in heap-profile parsing tools.
We track the ideal reported (to the end user) number of bytes more carefully
inside core jemalloc. When dumping heap profiles, insteading of outputting our
counts directly, we output counts that will cause parsing tools to give a result
close to the value we want.
We retain the old version as an opt setting, to let users who are tracking
values on a per-component basis to keep their metrics stable until they decide
to switch.
2020-08-05 18:33:55 -07:00
David Goldblatt
81c2f841e5
Add a simple utility to detect profiling bias.
2020-08-05 18:33:55 -07:00
Yinan Zhang
978f830ee3
Add batch allocation API
2020-07-31 09:16:50 -07:00
Yinan Zhang
c6f59e9bb4
Add surplus reading API for thread event lookahead
2020-07-31 09:16:50 -07:00
Yinan Zhang
f805468957
Add zero option to arena batch allocation
2020-07-31 09:16:50 -07:00
Yinan Zhang
49e5c2fe7d
Add batch allocation from fresh slabs
2020-07-31 09:16:50 -07:00
Yinan Zhang
2bb8060d57
Add empty test and concat for typed list
2020-07-31 09:16:50 -07:00
Yinan Zhang
f28cc2bc87
Extract bin shard selection out of bin locking
2020-07-31 09:16:50 -07:00
David Goldblatt
ddb8dc4ad0
FB: Add range iteration support.
2020-07-30 15:25:23 -07:00
David Goldblatt
ceee823519
Add flat_bitmap.
...
The flat_bitmap module offers an extended API, at the cost of decreased
performance in the case of very large bitmaps.
2020-07-30 15:25:23 -07:00
David Goldblatt
22da836094
bit_util: Add fls_ functions; "find last set".
...
These simplify a lot of the bit_util module, which had grown bits and pieces of
this functionality across a variety of places over the years.
While we're here, kill off BIT_UTIL_INLINE and don't do reentrancy testing for
bit_util.
2020-07-30 15:25:23 -07:00
David Goldblatt
1ed0288d9c
bit_util: Change ffs functions indexing.
...
Making these 0-based instead of 1-based makes calling code simpler and will be
more consistent with functions introduced in subsequent diffs.
2020-07-30 15:25:23 -07:00
David Goldblatt
6107857b7b
PA->PAC: Move in PAI implementation.
2020-07-09 13:41:04 -07:00
David Goldblatt
6041aaba97
PA -> PAC: Move in destruction functions.
2020-07-09 13:41:04 -07:00
David Goldblatt
471eb5913c
PAC: Move in decay rate setting.
2020-07-09 13:41:04 -07:00
David Goldblatt
6a2774719f
PA->PAC: Move in decay functions.
2020-07-09 13:41:04 -07:00
David Goldblatt
4ee75be3a3
PA -> PAC: Move in decay_purge enum.
2020-07-09 13:41:04 -07:00
David Goldblatt
72435b0aba
PA->PAC: Make extent.c forget about PA.
2020-07-09 13:41:04 -07:00
David Goldblatt
dee5d1c42d
PA->PAC: Move in extent_sn.
2020-07-09 13:41:04 -07:00
David Goldblatt
7391382349
PA->PAC: Move in stats.
2020-07-09 13:41:04 -07:00
David Goldblatt
db211eefbf
PAC: Move in decay.
2020-07-09 13:41:04 -07:00
David Goldblatt
c81e389996
PAC: Move in ecache_grow.
2020-07-09 13:41:04 -07:00
David Goldblatt
65803171a7
PAC: move in emap
2020-07-09 13:41:04 -07:00
David Goldblatt
7efcb946c4
PAC: Add an init function.
2020-07-09 13:41:04 -07:00
David Goldblatt
722652222a
PAC: Move in edata_cache accesses.
2020-07-09 13:41:04 -07:00
David Goldblatt
777b0ba965
Add PAC: Page allocator classic.
...
For now, this is just a stub containing the ecaches, with no surrounding code
changed. Eventually all the core allocator bits will be moved in, in the
subsequent stack of commits.
2020-07-09 13:41:04 -07:00
David Goldblatt
1b5f632e0f
Introduce PAI: Page allocator interface
2020-07-09 13:41:04 -07:00
David Goldblatt
3cf19c6e5e
atomic: add atomic_load_sub_store
2020-07-09 13:41:04 -07:00
David Goldblatt
ae541d3fab
Edata: Reserve some space for hugepages.
2020-07-08 13:20:59 -07:00
David Goldblatt
392f645f4d
Edata: split up different list linkage uses.
2020-07-08 13:20:59 -07:00
David Goldblatt
129b727058
Add typed-list module.
...
This gives some named convenience wrappers.
2020-07-08 13:20:59 -07:00
David Carlier
00f06c9beb
enabling mpss on solaris/illumos.
...
reusing slighty linux configuration as possible, aligning the
address range to HUGEPAGE.
2020-07-06 09:59:10 -07:00
Yinan Zhang
c2e7a06392
No need to intercept prof_dump_header() in tests
2020-06-29 14:27:50 -07:00
Yinan Zhang
f58ebdff7a
Generalize prof_cnt_all() for testing
2020-06-29 14:27:50 -07:00
Yinan Zhang
d4259ea53b
Simplify signatures for prof dump functions
2020-06-29 14:27:50 -07:00
Yinan Zhang
1f5fe3a3e3
Pass write callback explicitly in prof_data
2020-06-29 14:27:50 -07:00
Yinan Zhang
dad821bb22
Move unwind to prof_sys
2020-06-29 14:27:50 -07:00
Yinan Zhang
d128efcb6a
Relocate a few prof utilities to the right modules
2020-06-29 14:27:50 -07:00
Yinan Zhang
4736fb4fc9
Move file handling logic in prof_data to prof_sys
2020-06-29 14:27:50 -07:00
Yinan Zhang
767a2e1790
Move file handling logic in prof to prof_sys
2020-06-29 14:27:50 -07:00
Yinan Zhang
03ae509f32
Create prof_sys module for reading system thread name
2020-06-29 14:27:50 -07:00
Yinan Zhang
adfd9d7b1d
Change tsdn to tsd for thread name allocation
2020-06-29 14:27:50 -07:00
Yinan Zhang
841af2b426
Move thread name handling to prof_data module
2020-06-29 14:27:50 -07:00
Yinan Zhang
8118056c03
Expose prof_data testing internals only in prof tests
2020-06-29 14:27:50 -07:00
Yinan Zhang
f43ac8543e
Correct prof header macro namings
2020-06-29 14:27:50 -07:00
Yinan Zhang
5d292b5660
Push error handling logic out of core dumping logic
2020-06-29 14:27:50 -07:00
Yinan Zhang
f541871f5d
Reduce prof dump buffer size in debug build
2020-06-29 14:27:50 -07:00
Yinan Zhang
354183b10d
Define prof dump buffer size centrally
2020-06-29 14:27:50 -07:00
Yinan Zhang
7455813e57
Make dump file writing replaceable in test
2020-06-29 14:27:50 -07:00
Yinan Zhang
21e44c45d9
Make maps file opening replaceable in test
2020-06-29 14:27:50 -07:00
Yinan Zhang
f307b25804
Only replace the dump file opening function in test
2020-06-29 14:27:50 -07:00
Yinan Zhang
d460333efb
Improve naming for prof system thread name option
2020-06-24 14:32:01 -07:00
David T. Goldblatt
25e43c6022
Witness: Make ranks an enum.
...
This lets us avoid having to increment a bunch of values manually every time we
add a new sort of lock.
2020-06-19 18:05:08 -07:00
Yinan Zhang
b7858abfc0
Expose prof testing internal functions
2020-06-19 09:16:51 -07:00
Jon Haslam
4aea743279
High Resolution Timestamps for Profiling
2020-06-15 12:12:49 -07:00
David Goldblatt
d82a164d0d
Add thread.peak.[read|reset] mallctls.
...
These can be used to track net allocator activity on a per-thread basis.
2020-06-11 13:54:22 -07:00
David Goldblatt
fe7108305a
Add peak_t, for tracking allocator net max.
2020-06-11 13:54:22 -07:00
Yinan Zhang
3e19ebd2ea
Add lock to protect prof last-N dumping
2020-06-09 17:03:05 -07:00
Yinan Zhang
857ebd3daf
Make edata pointer on prof recent record an atomic fence
2020-06-09 17:03:05 -07:00
David Goldblatt
6cdac3c573
Tcache: Make flush fractions configurable.
2020-05-16 13:34:23 -07:00
David Goldblatt
ee72bf1cfd
Tcache: Add tcache gc delay option.
...
This can reduce flushing frequency for small size classes.
2020-05-16 13:34:23 -07:00
David Goldblatt
d338dd45d7
Tcache: Make incremental gc bytes configurable.
2020-05-16 13:34:23 -07:00
David Goldblatt
ec0b579563
Tcache: Privatize opt_lg_tcache_max default.
2020-05-16 13:34:23 -07:00
David Goldblatt
10b96f6351
Tcache: Remove some unused gc constants.
2020-05-16 13:34:23 -07:00
David Goldblatt
181093173d
Tcache: make slot sizing configurable.
2020-05-16 13:34:23 -07:00
David Goldblatt
b58dea8d1b
Cache bin: expose ncached_max publicly.
2020-05-16 13:34:23 -07:00
David Goldblatt
634afc4124
Tcache: Make size computation configurable.
2020-05-16 13:34:23 -07:00
Brooks Davis
27f29e424b
LQ_QUANTUM should be 4 on mips64 hardware.
...
This matches the ABI stack alignment requirements.
2020-05-14 10:30:37 -07:00
David Goldblatt
eda9c2858f
Edata: zero stack edatas before initializing.
...
This avoids some UB. No compilers take advantage of it for now, but no sense in
tempting fate.
2020-05-14 10:30:20 -07:00
Yinan Zhang
dcea2c0f8b
Get rid of TSD -> thread event dependency
2020-05-12 09:16:16 -07:00
Yinan Zhang
b06dfb9ccc
Push event handlers to constituent modules
2020-05-12 09:16:16 -07:00
Yinan Zhang
abd4674931
Extract out per event postponed wait time fetching
2020-05-12 09:16:16 -07:00
Yinan Zhang
f72014d097
Only compute thread event threshold once per trigger
2020-05-12 09:16:16 -07:00
Yinan Zhang
6de77799de
Move thread event wait time update to local
2020-05-12 09:16:16 -07:00
Yinan Zhang
733ae918f0
Extract out per event new wait time fetching
2020-05-12 09:16:16 -07:00
Yinan Zhang
1e2524e15a
Do not reset sample wait time when re-initing tdata
2020-05-12 09:16:16 -07:00
Yinan Zhang
855d20f6f3
Remove outdated comments in thread event
2020-05-12 09:16:16 -07:00
Yinan Zhang
fc052ff728
Migrate counter to use locked int
2020-05-12 08:23:15 -07:00
Yinan Zhang
b543c20a94
Minor update to locked int
2020-05-12 08:23:15 -07:00
Yinan Zhang
f533ab6da6
Add forking handling for stats
2020-05-11 15:35:06 -07:00
Yinan Zhang
4d970f8bfc
Add forking handling for counter module
2020-05-11 15:35:06 -07:00
Yinan Zhang
2097e1945b
Unify write callback signature
2020-05-11 14:51:24 -07:00
Yinan Zhang
fef9abdcc0
Cleanup tcache allocation logic
...
The logic in tcache allocation no longer involves profiling or
filling.
2020-05-11 12:24:56 -07:00
Yinan Zhang
e6cb6919c0
Consolidate prof inline function headers
...
The prof inline functions are no longer involved in a circular
dependency, so consolidate the two headers into one.
2020-05-11 12:24:56 -07:00
Yinan Zhang
d454af90f1
Remove unused prof_accum field from arena
2020-05-11 12:24:56 -07:00
Yinan Zhang
8be5584494
Initialize prof idump counter once rather than once per arena
2020-05-11 12:24:56 -07:00
Yinan Zhang
e10e5059e8
Make prof_idump_accum() non-inline
2020-05-11 12:24:56 -07:00
Yinan Zhang
039bfd4e30
Do not rollback prof idump counter in arena_prof_promote()
2020-05-11 12:24:56 -07:00
David Goldblatt
46471ea327
SC: Name the max lookup constant.
2020-05-04 12:27:07 -07:00
David Goldblatt
79dd0c04ed
SC: Simplify SC_NPSIZES computation.
...
Rather than taking all the sizes and subtracting out those that don't fit, we
instead just add up all the ones that do.
2020-05-04 12:27:07 -07:00
David Goldblatt
4f8efba824
TSD: Make rtree_ctx a slow-path field.
...
Performance-sensitive users will use sized deallocation facilities, so that
actually touching the rtree_ctx is unnecessary. We make it the last element of
the slow data, so that it is for practical purposes almost-fast.
2020-04-14 15:20:19 -07:00
David Goldblatt
cd29ebefd0
Tcache: treat small and large cache bins uniformly
2020-04-14 15:20:19 -07:00
David Goldblatt
a13fbad374
Tcache: split up fast and slow path data.
2020-04-14 15:20:19 -07:00
David Goldblatt
7099c66205
Arena: fill in terms of cache_bins.
2020-04-14 15:20:19 -07:00
David Goldblatt
40e7aed59e
TSD: Move in some of the tcache fields.
...
We had put these in the tcache for cache optimization reasons. After the
previous diff, these no longer apply.
2020-04-14 15:20:19 -07:00
David Goldblatt
58a00df238
TSD: Put all fast-path data together.
2020-04-14 15:20:19 -07:00
David Goldblatt
877af247a8
QL, QR: Add documentation.
2020-04-11 10:32:11 -07:00
David Goldblatt
79ae7f9211
Rtree: Remove the per-field accessors.
...
We instead split things into "edata" and "metadata".
2020-04-10 13:12:47 -07:00
David Goldblatt
bb6a418523
Emap: Drop szind/slab splitting parameters.
...
After the previous diff, these are constants.
2020-04-10 13:12:47 -07:00
David Goldblatt
50289750b3
Extent: Remove szind/slab knowledge.
2020-04-10 13:12:47 -07:00
David Goldblatt
dc26b30094
Rtree: Clean up compact/non-compact split.
2020-04-10 13:12:47 -07:00
David Goldblatt
93b99dd140
Extent: Stop passing an edata_cache everywhere.
...
We already pass the pa_shard_t around everywhere; we can just use that.
2020-04-10 13:12:47 -07:00
David Goldblatt
11c47cb133
Extent: Take "bool zero" over "bool *zero".
2020-04-10 13:12:47 -07:00
David Goldblatt
1a1124462e
PA: Take zero as a bool rather than as a bool *.
...
Now that we've moved junking to a higher level of the allocation stack, we don't
care about this performance optimization (which only occurred in debug modes).
2020-04-10 13:12:47 -07:00
David Goldblatt
294b276fc7
PA: Parameterize emap. Move emap_global to arena.
...
This lets us test the PA module without interfering with the global emap used by
the real allocator (the one not under test).
2020-04-10 13:12:47 -07:00
David Goldblatt
f730577277
Eset: Parameterize last globals accesses.
...
I.e. opt_retain and maps_coalesce.
2020-04-10 13:12:47 -07:00
David Goldblatt
7bb6e2dc0d
Eset: take opt_lg_max_active_fit as a parameter.
...
This breaks its dependence on the global.
2020-04-10 13:12:47 -07:00
David Goldblatt
883ab327cc
Emap: Move out last edata state touching.
2020-04-10 13:12:47 -07:00
David Goldblatt
12eb888e54
Edata: Add a ranged bit.
...
We steal the dumpable bit, which we ended up not needing.
2020-04-10 13:12:47 -07:00
David Goldblatt
bd4fdf295e
Rtree: Pull leaf contents into their own struct.
2020-04-10 13:12:47 -07:00
David Goldblatt
faec7219b2
PA: Move in decay initialization.
2020-04-10 13:12:47 -07:00
David Goldblatt
45671e4a27
PA: Move in retain growth limit setting.
2020-04-10 13:12:47 -07:00
David Goldblatt
daefde88fe
PA: Move in mutex stats reading.
2020-04-10 13:12:47 -07:00
David Goldblatt
07675840a5
PA: Move in some more internals accesses.
2020-04-10 13:12:47 -07:00
David Goldblatt
238f3c7430
PA: Move in full stats merging.
2020-04-10 13:12:47 -07:00
David Goldblatt
81c6027592
Arena stats: Give it its own "mapped".
...
This distinguishes it from the PA mapped stat, which is now named "pa_mapped" to
avoid confusion. The (derived) arena stat includes base memory, and the PA stat
is no longer partially derived.
2020-04-10 13:12:47 -07:00
David Goldblatt
506d907e40
PA: Move in basic stats merging.
2020-04-10 13:12:47 -07:00
David Goldblatt
f29f6090f5
PA: Add pa_extra.c and put PA forking there.
2020-04-10 13:12:47 -07:00
David Goldblatt
565045ef71
Arena: Make more derived stats non-atomic/locked.
2020-04-10 13:12:47 -07:00
David Goldblatt
d0c43217b5
Arena stats: Move retained to PA, use plain ints.
...
Retained is a property of the allocated pages. The derived fields no longer
require any locking; they're computed on demand.
2020-04-10 13:12:47 -07:00
David Goldblatt
e2cf3fb1a3
PA: Move in all modifications of mapped.
2020-04-10 13:12:47 -07:00
David Goldblatt
436789ad96
PA: Make mapped stat atomic.
...
We always have atomic_zu_t, and mapped/unmapped transitions are always expensive
enough that trying to piggyback on a lock is a waste of time.
2020-04-10 13:12:47 -07:00
David Goldblatt
3c28aa6f17
PA: Move edata_avail stat in, make it non-atomic.
2020-04-10 13:12:47 -07:00
David Goldblatt
f6bfa3dcca
Move extent stats to the PA module.
...
While we're at it, make them non-atomic -- they are purely derived statistics
(and in fact aren't even in the arena_t or pa_shard_t).
2020-04-10 13:12:47 -07:00
David Goldblatt
527dd4cdb8
PA: Move in nactive counter.
2020-04-10 13:12:47 -07:00
David Goldblatt
c075fd0bcb
PA: Minor cleanups and comment fixes.
2020-04-10 13:12:47 -07:00
David Goldblatt
46a9d7fc0b
PA: Move in rest of purging.
2020-04-10 13:12:47 -07:00
David Goldblatt
2d6eec7b5c
PA: Move in decay-all pathway.
2020-04-10 13:12:47 -07:00
David Goldblatt
65698b7f2e
PA: Remove public visibility of some internals.
2020-04-10 13:12:47 -07:00
David Goldblatt
f012c43be0
PA: Move in decay_to_limit
2020-04-10 13:12:47 -07:00
David Goldblatt
3034f4a508
PA: Move in decay_stashed.
2020-04-10 13:12:47 -07:00
David Goldblatt
aef28b2f8f
PA: Move in stash_decayed.
2020-04-10 13:12:47 -07:00
David Goldblatt
71fc0dc968
PA: Move in remaining page allocation functions.
2020-04-10 13:12:47 -07:00
David Goldblatt
74958567a4
PA: have expand take sizes instead of new usize.
...
This avoids involving usize, which makes some of the stats modifications more
intuitively correct.
2020-04-10 13:12:47 -07:00