Yinan Zhang
dad821bb22
Move unwind to prof_sys
2020-06-29 14:27:50 -07:00
Yinan Zhang
d128efcb6a
Relocate a few prof utilities to the right modules
2020-06-29 14:27:50 -07:00
Yinan Zhang
4736fb4fc9
Move file handling logic in prof_data to prof_sys
2020-06-29 14:27:50 -07:00
Yinan Zhang
767a2e1790
Move file handling logic in prof to prof_sys
2020-06-29 14:27:50 -07:00
Yinan Zhang
03ae509f32
Create prof_sys module for reading system thread name
2020-06-29 14:27:50 -07:00
Yinan Zhang
adfd9d7b1d
Change tsdn to tsd for thread name allocation
2020-06-29 14:27:50 -07:00
Yinan Zhang
841af2b426
Move thread name handling to prof_data module
2020-06-29 14:27:50 -07:00
Yinan Zhang
8118056c03
Expose prof_data testing internals only in prof tests
2020-06-29 14:27:50 -07:00
Yinan Zhang
f43ac8543e
Correct prof header macro namings
2020-06-29 14:27:50 -07:00
Yinan Zhang
5d292b5660
Push error handling logic out of core dumping logic
2020-06-29 14:27:50 -07:00
Yinan Zhang
f541871f5d
Reduce prof dump buffer size in debug build
2020-06-29 14:27:50 -07:00
Yinan Zhang
354183b10d
Define prof dump buffer size centrally
2020-06-29 14:27:50 -07:00
Yinan Zhang
7455813e57
Make dump file writing replaceable in test
2020-06-29 14:27:50 -07:00
Yinan Zhang
21e44c45d9
Make maps file opening replaceable in test
2020-06-29 14:27:50 -07:00
Yinan Zhang
f307b25804
Only replace the dump file opening function in test
2020-06-29 14:27:50 -07:00
Yinan Zhang
d460333efb
Improve naming for prof system thread name option
2020-06-24 14:32:01 -07:00
David T. Goldblatt
25e43c6022
Witness: Make ranks an enum.
...
This lets us avoid having to increment a bunch of values manually every time we
add a new sort of lock.
2020-06-19 18:05:08 -07:00
Yinan Zhang
b7858abfc0
Expose prof testing internal functions
2020-06-19 09:16:51 -07:00
Jon Haslam
4aea743279
High Resolution Timestamps for Profiling
2020-06-15 12:12:49 -07:00
David Goldblatt
d82a164d0d
Add thread.peak.[read|reset] mallctls.
...
These can be used to track net allocator activity on a per-thread basis.
2020-06-11 13:54:22 -07:00
David Goldblatt
fe7108305a
Add peak_t, for tracking allocator net max.
2020-06-11 13:54:22 -07:00
Yinan Zhang
3e19ebd2ea
Add lock to protect prof last-N dumping
2020-06-09 17:03:05 -07:00
Yinan Zhang
857ebd3daf
Make edata pointer on prof recent record an atomic fence
2020-06-09 17:03:05 -07:00
David Goldblatt
6cdac3c573
Tcache: Make flush fractions configurable.
2020-05-16 13:34:23 -07:00
David Goldblatt
ee72bf1cfd
Tcache: Add tcache gc delay option.
...
This can reduce flushing frequency for small size classes.
2020-05-16 13:34:23 -07:00
David Goldblatt
d338dd45d7
Tcache: Make incremental gc bytes configurable.
2020-05-16 13:34:23 -07:00
David Goldblatt
ec0b579563
Tcache: Privatize opt_lg_tcache_max default.
2020-05-16 13:34:23 -07:00
David Goldblatt
10b96f6351
Tcache: Remove some unused gc constants.
2020-05-16 13:34:23 -07:00
David Goldblatt
181093173d
Tcache: make slot sizing configurable.
2020-05-16 13:34:23 -07:00
David Goldblatt
b58dea8d1b
Cache bin: expose ncached_max publicly.
2020-05-16 13:34:23 -07:00
David Goldblatt
634afc4124
Tcache: Make size computation configurable.
2020-05-16 13:34:23 -07:00
Brooks Davis
27f29e424b
LQ_QUANTUM should be 4 on mips64 hardware.
...
This matches the ABI stack alignment requirements.
2020-05-14 10:30:37 -07:00
David Goldblatt
eda9c2858f
Edata: zero stack edatas before initializing.
...
This avoids some UB. No compilers take advantage of it for now, but no sense in
tempting fate.
2020-05-14 10:30:20 -07:00
Yinan Zhang
dcea2c0f8b
Get rid of TSD -> thread event dependency
2020-05-12 09:16:16 -07:00
Yinan Zhang
b06dfb9ccc
Push event handlers to constituent modules
2020-05-12 09:16:16 -07:00
Yinan Zhang
abd4674931
Extract out per event postponed wait time fetching
2020-05-12 09:16:16 -07:00
Yinan Zhang
f72014d097
Only compute thread event threshold once per trigger
2020-05-12 09:16:16 -07:00
Yinan Zhang
6de77799de
Move thread event wait time update to local
2020-05-12 09:16:16 -07:00
Yinan Zhang
733ae918f0
Extract out per event new wait time fetching
2020-05-12 09:16:16 -07:00
Yinan Zhang
1e2524e15a
Do not reset sample wait time when re-initing tdata
2020-05-12 09:16:16 -07:00
Yinan Zhang
855d20f6f3
Remove outdated comments in thread event
2020-05-12 09:16:16 -07:00
Yinan Zhang
fc052ff728
Migrate counter to use locked int
2020-05-12 08:23:15 -07:00
Yinan Zhang
b543c20a94
Minor update to locked int
2020-05-12 08:23:15 -07:00
Yinan Zhang
f533ab6da6
Add forking handling for stats
2020-05-11 15:35:06 -07:00
Yinan Zhang
4d970f8bfc
Add forking handling for counter module
2020-05-11 15:35:06 -07:00
Yinan Zhang
2097e1945b
Unify write callback signature
2020-05-11 14:51:24 -07:00
Yinan Zhang
fef9abdcc0
Cleanup tcache allocation logic
...
The logic in tcache allocation no longer involves profiling or
filling.
2020-05-11 12:24:56 -07:00
Yinan Zhang
e6cb6919c0
Consolidate prof inline function headers
...
The prof inline functions are no longer involved in a circular
dependency, so consolidate the two headers into one.
2020-05-11 12:24:56 -07:00
Yinan Zhang
d454af90f1
Remove unused prof_accum field from arena
2020-05-11 12:24:56 -07:00
Yinan Zhang
8be5584494
Initialize prof idump counter once rather than once per arena
2020-05-11 12:24:56 -07:00
Yinan Zhang
e10e5059e8
Make prof_idump_accum() non-inline
2020-05-11 12:24:56 -07:00
Yinan Zhang
039bfd4e30
Do not rollback prof idump counter in arena_prof_promote()
2020-05-11 12:24:56 -07:00
David Goldblatt
46471ea327
SC: Name the max lookup constant.
2020-05-04 12:27:07 -07:00
David Goldblatt
79dd0c04ed
SC: Simplify SC_NPSIZES computation.
...
Rather than taking all the sizes and subtracting out those that don't fit, we
instead just add up all the ones that do.
2020-05-04 12:27:07 -07:00
David Goldblatt
4f8efba824
TSD: Make rtree_ctx a slow-path field.
...
Performance-sensitive users will use sized deallocation facilities, so that
actually touching the rtree_ctx is unnecessary. We make it the last element of
the slow data, so that it is for practical purposes almost-fast.
2020-04-14 15:20:19 -07:00
David Goldblatt
cd29ebefd0
Tcache: treat small and large cache bins uniformly
2020-04-14 15:20:19 -07:00
David Goldblatt
a13fbad374
Tcache: split up fast and slow path data.
2020-04-14 15:20:19 -07:00
David Goldblatt
7099c66205
Arena: fill in terms of cache_bins.
2020-04-14 15:20:19 -07:00
David Goldblatt
40e7aed59e
TSD: Move in some of the tcache fields.
...
We had put these in the tcache for cache optimization reasons. After the
previous diff, these no longer apply.
2020-04-14 15:20:19 -07:00
David Goldblatt
58a00df238
TSD: Put all fast-path data together.
2020-04-14 15:20:19 -07:00
David Goldblatt
877af247a8
QL, QR: Add documentation.
2020-04-11 10:32:11 -07:00
David Goldblatt
79ae7f9211
Rtree: Remove the per-field accessors.
...
We instead split things into "edata" and "metadata".
2020-04-10 13:12:47 -07:00
David Goldblatt
bb6a418523
Emap: Drop szind/slab splitting parameters.
...
After the previous diff, these are constants.
2020-04-10 13:12:47 -07:00
David Goldblatt
50289750b3
Extent: Remove szind/slab knowledge.
2020-04-10 13:12:47 -07:00
David Goldblatt
dc26b30094
Rtree: Clean up compact/non-compact split.
2020-04-10 13:12:47 -07:00
David Goldblatt
93b99dd140
Extent: Stop passing an edata_cache everywhere.
...
We already pass the pa_shard_t around everywhere; we can just use that.
2020-04-10 13:12:47 -07:00
David Goldblatt
11c47cb133
Extent: Take "bool zero" over "bool *zero".
2020-04-10 13:12:47 -07:00
David Goldblatt
1a1124462e
PA: Take zero as a bool rather than as a bool *.
...
Now that we've moved junking to a higher level of the allocation stack, we don't
care about this performance optimization (which only occurred in debug modes).
2020-04-10 13:12:47 -07:00
David Goldblatt
294b276fc7
PA: Parameterize emap. Move emap_global to arena.
...
This lets us test the PA module without interfering with the global emap used by
the real allocator (the one not under test).
2020-04-10 13:12:47 -07:00
David Goldblatt
f730577277
Eset: Parameterize last globals accesses.
...
I.e. opt_retain and maps_coalesce.
2020-04-10 13:12:47 -07:00
David Goldblatt
7bb6e2dc0d
Eset: take opt_lg_max_active_fit as a parameter.
...
This breaks its dependence on the global.
2020-04-10 13:12:47 -07:00
David Goldblatt
883ab327cc
Emap: Move out last edata state touching.
2020-04-10 13:12:47 -07:00
David Goldblatt
12eb888e54
Edata: Add a ranged bit.
...
We steal the dumpable bit, which we ended up not needing.
2020-04-10 13:12:47 -07:00
David Goldblatt
bd4fdf295e
Rtree: Pull leaf contents into their own struct.
2020-04-10 13:12:47 -07:00
David Goldblatt
faec7219b2
PA: Move in decay initialization.
2020-04-10 13:12:47 -07:00
David Goldblatt
45671e4a27
PA: Move in retain growth limit setting.
2020-04-10 13:12:47 -07:00
David Goldblatt
daefde88fe
PA: Move in mutex stats reading.
2020-04-10 13:12:47 -07:00
David Goldblatt
07675840a5
PA: Move in some more internals accesses.
2020-04-10 13:12:47 -07:00
David Goldblatt
238f3c7430
PA: Move in full stats merging.
2020-04-10 13:12:47 -07:00
David Goldblatt
81c6027592
Arena stats: Give it its own "mapped".
...
This distinguishes it from the PA mapped stat, which is now named "pa_mapped" to
avoid confusion. The (derived) arena stat includes base memory, and the PA stat
is no longer partially derived.
2020-04-10 13:12:47 -07:00
David Goldblatt
506d907e40
PA: Move in basic stats merging.
2020-04-10 13:12:47 -07:00
David Goldblatt
f29f6090f5
PA: Add pa_extra.c and put PA forking there.
2020-04-10 13:12:47 -07:00
David Goldblatt
565045ef71
Arena: Make more derived stats non-atomic/locked.
2020-04-10 13:12:47 -07:00
David Goldblatt
d0c43217b5
Arena stats: Move retained to PA, use plain ints.
...
Retained is a property of the allocated pages. The derived fields no longer
require any locking; they're computed on demand.
2020-04-10 13:12:47 -07:00
David Goldblatt
e2cf3fb1a3
PA: Move in all modifications of mapped.
2020-04-10 13:12:47 -07:00
David Goldblatt
436789ad96
PA: Make mapped stat atomic.
...
We always have atomic_zu_t, and mapped/unmapped transitions are always expensive
enough that trying to piggyback on a lock is a waste of time.
2020-04-10 13:12:47 -07:00
David Goldblatt
3c28aa6f17
PA: Move edata_avail stat in, make it non-atomic.
2020-04-10 13:12:47 -07:00
David Goldblatt
f6bfa3dcca
Move extent stats to the PA module.
...
While we're at it, make them non-atomic -- they are purely derived statistics
(and in fact aren't even in the arena_t or pa_shard_t).
2020-04-10 13:12:47 -07:00
David Goldblatt
527dd4cdb8
PA: Move in nactive counter.
2020-04-10 13:12:47 -07:00
David Goldblatt
c075fd0bcb
PA: Minor cleanups and comment fixes.
2020-04-10 13:12:47 -07:00
David Goldblatt
46a9d7fc0b
PA: Move in rest of purging.
2020-04-10 13:12:47 -07:00
David Goldblatt
2d6eec7b5c
PA: Move in decay-all pathway.
2020-04-10 13:12:47 -07:00
David Goldblatt
65698b7f2e
PA: Remove public visibility of some internals.
2020-04-10 13:12:47 -07:00
David Goldblatt
f012c43be0
PA: Move in decay_to_limit
2020-04-10 13:12:47 -07:00
David Goldblatt
3034f4a508
PA: Move in decay_stashed.
2020-04-10 13:12:47 -07:00
David Goldblatt
aef28b2f8f
PA: Move in stash_decayed.
2020-04-10 13:12:47 -07:00
David Goldblatt
71fc0dc968
PA: Move in remaining page allocation functions.
2020-04-10 13:12:47 -07:00
David Goldblatt
74958567a4
PA: have expand take sizes instead of new usize.
...
This avoids involving usize, which makes some of the stats modifications more
intuitively correct.
2020-04-10 13:12:47 -07:00
David Goldblatt
5bcc2c2ab9
PA: Have expand take szind and slab.
...
This isn't really necessary, but having a uniform API will help us later.
2020-04-10 13:12:47 -07:00
David Goldblatt
0880c2ab97
PA: Have large expands use it.
2020-04-10 13:12:47 -07:00
David Goldblatt
9f93625c14
PA: Move in arena large allocation functionality.
2020-04-10 13:12:47 -07:00
David Goldblatt
7624043a41
PA: Add ehook-getting support.
2020-04-10 13:12:47 -07:00
David Goldblatt
eba35e2e48
Remove extent knowledge of arena.
2020-04-10 13:12:47 -07:00
David Goldblatt
e77f47a85a
Move arena decay getters to PA.
2020-04-10 13:12:47 -07:00
David Goldblatt
f77cec311e
Decay: Take current time as an argument.
...
This better facilitates testing.
2020-04-10 13:12:47 -07:00
David Goldblatt
d1d7e1076b
Decay: move in some background_thread accesses.
2020-04-10 13:12:47 -07:00
David Goldblatt
cdb916ed3f
Decay: Add comments for the public API.
2020-04-10 13:12:47 -07:00
David Goldblatt
8f2193dc8d
Decay: Move in arena decay functions.
2020-04-10 13:12:47 -07:00
David Goldblatt
7b62885476
Introduce decay module and put decay objects in PA
2020-04-10 13:12:47 -07:00
David Goldblatt
497836dbc8
Arena stats: mark edata_avail as derived.
...
The true number is in the edata_cache itself.
2020-04-10 13:12:47 -07:00
David Goldblatt
3192d6b77d
Extents: Have extent_dalloc_gap take ehooks.
...
We're almost to the point where the extent code doesn't know about arenas at
all. In that world, we shouldn't pull them out of the arena.
2020-04-10 13:12:47 -07:00
David Goldblatt
22a0a7b93a
Move arena_decay_extent to extent module.
2020-04-10 13:12:47 -07:00
David Goldblatt
70d12ffa05
PA: Move mapped into pa stats.
2020-04-10 13:12:47 -07:00
David Goldblatt
6ca918d0cf
PA: Add a stats comment.
2020-04-10 13:12:47 -07:00
David Goldblatt
ce8c0d6c09
PA: Move in arena extent_sn counter.
...
Just another step towards making PA self-contained.
2020-04-10 13:12:47 -07:00
David Goldblatt
1ad368c8b7
PA: Move in decay stats.
2020-04-10 13:12:47 -07:00
David Goldblatt
356aaa7dc6
Introduce lockedint module.
...
This pulls out the various abstractions where some stats counter is sometimes an
atomic, sometimes a plain variable, sometimes always protected by a lock,
sometimes protected by reads but not writes, etc. With this change, these cases
are treated consistently, and access patterns tagged.
In the process, we fix a few missed-update bugs (where one caller assumes
"protected-by-a-lock" semantics and another does not).
2020-04-10 13:12:47 -07:00
David Goldblatt
acd0bf6a26
PA: move in ecache_grow.
2020-04-10 13:12:47 -07:00
David Goldblatt
32cb7c2f0b
PA: Add a stats type.
2020-04-10 13:12:47 -07:00
David Goldblatt
688fb3eb89
PA: Move in the arena edata_cache.
2020-04-10 13:12:47 -07:00
David Goldblatt
8433ad84ea
PA: move in shard initialization.
2020-04-10 13:12:47 -07:00
David Goldblatt
a24faed569
PA: Move in the ecache_t objects.
2020-04-10 13:12:47 -07:00
David Goldblatt
585f925055
Move cache index randomization out of extent.
...
This is logically at a higher level of the stack; extent should just allocate
things at the page-level; it shouldn't care exactly why the callers wants a
given number of pages.
2020-04-10 13:12:47 -07:00
David Goldblatt
12be9f5727
Add a stub PA module -- a page allocator.
2020-04-10 13:12:47 -07:00
Yinan Zhang
c4e9ea8cc6
Get rid of locks in prof recent test
2020-04-07 17:22:24 -07:00
Yinan Zhang
2deabac079
Get rid of custom iterator for last-N records
2020-04-07 17:22:24 -07:00
Yinan Zhang
a5ddfa7d91
Use ql for prof last-N list
2020-04-07 17:22:24 -07:00
Yinan Zhang
ce17af4221
Better structure ql module
2020-04-06 09:50:27 -07:00
Yinan Zhang
4b66297ea0
Add move constructor to ql module
2020-04-06 09:50:27 -07:00
Yinan Zhang
a62b7ed928
Add emptiness checking to ql module
2020-04-06 09:50:27 -07:00
Yinan Zhang
1dd24ca6d2
Add rotate functionality to ql module
2020-04-06 09:50:27 -07:00
Yinan Zhang
0dc95a882f
Add concat and split functionality to ql module
2020-04-06 09:50:27 -07:00
Yinan Zhang
1ad06aa53b
deduplicate insert and delete logic in qr module
2020-04-06 09:50:27 -07:00
Yinan Zhang
c9d56cddf2
Optimize meld in qr module
...
The goal of `qr_meld()` is to change the following four fields
`(a->prev, a->prev->next, b->prev, b->prev->next)` from the values
`(a->prev, a, b->prev, b)` to `(b->prev, b, a->prev, a)`.
This commit changes
```
a->prev->next = b;
b->prev->next = a;
temp = a->prev;
a->prev = b->prev;
b->prev = temp;
```
to
```
temp = a->prev;
a->prev = b->prev;
b->prev = temp;
a->prev->next = a;
b->prev->next = b;
```
The benefit is that we can use `b->prev->next` for `temp`, and so
there's no need to pass in `a_type`.
The restriction is that `b` cannot be a `qr_next()` macro, so users
of `qr_meld()` must pay attention. (Before this change, neither `a`
nor `b` could be a `qr_next()` macro.)
2020-04-06 09:50:27 -07:00
Yinan Zhang
f9aad7a49b
Add piping API to buffered writer
2020-04-01 09:41:20 -07:00
Yinan Zhang
09cd79495f
Encapsulate buffer allocation failure in buffered writer
2020-04-01 09:41:20 -07:00
David Goldblatt
3b4a03b92b
Mac: don't declare system functions as nothrow.
...
This contradicts the system headers, which can lead to breakages.
2020-03-26 14:11:24 -07:00
Yinan Zhang
2256ef8961
Add option to fetch system thread name on each prof sample
2020-03-24 21:39:57 -07:00
Yinan Zhang
a5780598b3
Remove thread_event_rollback()
2020-03-12 13:55:00 -07:00
Yinan Zhang
ba783b3a0f
Remove prof -> thread_event dependency
2020-03-12 13:55:00 -07:00
Yinan Zhang
441d88d1c7
Rewrite profiling thread event
2020-03-12 13:55:00 -07:00
David Goldblatt
99b1291d17
Edata cache: add edata_cache_small_t.
...
This can be used to amortize the synchronization costs of edata_cache accesses.
2020-03-12 11:58:09 -07:00
David Goldblatt
92485032b2
Cache bin: improve comments.
2020-03-12 11:54:19 -07:00
David Goldblatt
d701a085c2
Fast path: allow low-water mark changes.
...
This lets us put more allocations on an "almost as fast" path after a flush.
This results in around a 4% reduction in malloc cycles in prod workloads
(corresponding to about a 0.1% reduction in overall cycles).
2020-03-12 11:54:19 -07:00
David Goldblatt
397da03865
Cache bin: rewrite to track more state.
...
With this, we track all of the empty, full, and low water states together. This
simplifies a lot of the tracking logic, since we now don't need the
cache_bin_info_t for state queries (except for some debugging).
2020-03-12 11:54:19 -07:00
David Goldblatt
0a2fcfac01
Tcache: Hold cache bin allocation explicitly.
2020-03-12 11:54:19 -07:00
David Goldblatt
d498a4bb08
Cache bin: Add an emptiness assertion.
2020-03-12 11:54:19 -07:00
David Goldblatt
6a7aa46ef7
Cache bin: Add a debug method for init checking.
2020-03-12 11:54:19 -07:00
David Goldblatt
370c1ea007
Cache bin: Write the unit test in terms of the API
...
I.e. stop allowing the unit test to have secret access to implementation
internals.
2020-03-12 11:54:19 -07:00
David Goldblatt
7f5ebd211c
Cache bin: set low-water internally.
2020-03-12 11:54:19 -07:00