project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,836 Commits 2 Branches 0 Tags 13 MiB
a795b19327
Commit Graph

1502 Commits

Author SHA1 Message Date
David Goldblatt
a13fbad374 Tcache: split up fast and slow path data. 2020-04-14 15:20:19 -07:00
David Goldblatt
7099c66205 Arena: fill in terms of cache_bins. 2020-04-14 15:20:19 -07:00
David Goldblatt
40e7aed59e TSD: Move in some of the tcache fields. 
We had put these in the tcache for cache optimization reasons.  After the
previous diff, these no longer apply.
 2020-04-14 15:20:19 -07:00
David Goldblatt
3589571bfd SC: use SC_LG_NGROUP instead of its value. 
This magic constant introduces inconsistencies.  We should be able to change its
value solely by adjusting the definition in the header.
 2020-04-13 10:01:30 -07:00
David Goldblatt
79ae7f9211 Rtree: Remove the per-field accessors. 
We instead split things into "edata" and "metadata".
 2020-04-10 13:12:47 -07:00
David Goldblatt
bb6a418523 Emap: Drop szind/slab splitting parameters. 
After the previous diff, these are constants.
 2020-04-10 13:12:47 -07:00
David Goldblatt
50289750b3 Extent: Remove szind/slab knowledge. 2020-04-10 13:12:47 -07:00
David Goldblatt
dc26b30094 Rtree: Clean up compact/non-compact split. 2020-04-10 13:12:47 -07:00
David Goldblatt
93b99dd140 Extent: Stop passing an edata_cache everywhere. 
We already pass the pa_shard_t around everywhere; we can just use that.
 2020-04-10 13:12:47 -07:00
David Goldblatt
a4759a1911 Ehooks: avoid touching arena_emap_global in tests. 
That breaks our ability to test custom emaps in isolation.
 2020-04-10 13:12:47 -07:00
David Goldblatt
11c47cb133 Extent: Take "bool zero" over "bool *zero". 2020-04-10 13:12:47 -07:00
David Goldblatt
1a1124462e PA: Take zero as a bool rather than as a bool *. 
Now that we've moved junking to a higher level of the allocation stack, we don't
care about this performance optimization (which only occurred in debug modes).
 2020-04-10 13:12:47 -07:00
David Goldblatt
294b276fc7 PA: Parameterize emap. Move emap_global to arena. 
This lets us test the PA module without interfering with the global emap used by
the real allocator (the one not under test).
 2020-04-10 13:12:47 -07:00
David Goldblatt
f730577277 Eset: Parameterize last globals accesses. 
I.e. opt_retain and maps_coalesce.
 2020-04-10 13:12:47 -07:00
David Goldblatt
7bb6e2dc0d Eset: take opt_lg_max_active_fit as a parameter. 
This breaks its dependence on the global.
 2020-04-10 13:12:47 -07:00
David Goldblatt
883ab327cc Emap: Move out last edata state touching. 2020-04-10 13:12:47 -07:00
David Goldblatt
0c96a2f03b Emap: Move out remaining edata modifications. 2020-04-10 13:12:47 -07:00
David Goldblatt
dfef0df71a Emap: Move edata modification out of emap_remap. 2020-04-10 13:12:47 -07:00
David Goldblatt
12eb888e54 Edata: Add a ranged bit. 
We steal the dumpable bit, which we ended up not needing.
 2020-04-10 13:12:47 -07:00
David Goldblatt
bd4fdf295e Rtree: Pull leaf contents into their own struct. 2020-04-10 13:12:47 -07:00
David Goldblatt
faec7219b2 PA: Move in decay initialization. 2020-04-10 13:12:47 -07:00
David Goldblatt
45671e4a27 PA: Move in retain growth limit setting. 2020-04-10 13:12:47 -07:00
David Goldblatt
daefde88fe PA: Move in mutex stats reading. 2020-04-10 13:12:47 -07:00
David Goldblatt
07675840a5 PA: Move in some more internals accesses. 2020-04-10 13:12:47 -07:00
David Goldblatt
238f3c7430 PA: Move in full stats merging. 2020-04-10 13:12:47 -07:00
David Goldblatt
81c6027592 Arena stats: Give it its own "mapped". 
This distinguishes it from the PA mapped stat, which is now named "pa_mapped" to
avoid confusion. The (derived) arena stat includes base memory, and the PA stat
is no longer partially derived.
 2020-04-10 13:12:47 -07:00
David Goldblatt
506d907e40 PA: Move in basic stats merging. 2020-04-10 13:12:47 -07:00
David Goldblatt
f29f6090f5 PA: Add pa_extra.c and put PA forking there. 2020-04-10 13:12:47 -07:00
David Goldblatt
8164fad404 Stats: Fix edata_cache size merging. 
Previously, we assigned to the output rather than incrementing it.
 2020-04-10 13:12:47 -07:00
David Goldblatt
565045ef71 Arena: Make more derived stats non-atomic/locked. 2020-04-10 13:12:47 -07:00
David Goldblatt
d0c43217b5 Arena stats: Move retained to PA, use plain ints. 
Retained is a property of the allocated pages.  The derived fields no longer
require any locking; they're computed on demand.
 2020-04-10 13:12:47 -07:00
David Goldblatt
e2cf3fb1a3 PA: Move in all modifications of mapped. 2020-04-10 13:12:47 -07:00
David Goldblatt
436789ad96 PA: Make mapped stat atomic. 
We always have atomic_zu_t, and mapped/unmapped transitions are always expensive
enough that trying to piggyback on a lock is a waste of time.
 2020-04-10 13:12:47 -07:00
David Goldblatt
3c28aa6f17 PA: Move edata_avail stat in, make it non-atomic. 2020-04-10 13:12:47 -07:00
David Goldblatt
f6bfa3dcca Move extent stats to the PA module. 
While we're at it, make them non-atomic -- they are purely derived statistics
(and in fact aren't even in the arena_t or pa_shard_t).
 2020-04-10 13:12:47 -07:00
David Goldblatt
527dd4cdb8 PA: Move in nactive counter. 2020-04-10 13:12:47 -07:00
David Goldblatt
c075fd0bcb PA: Minor cleanups and comment fixes. 2020-04-10 13:12:47 -07:00
David Goldblatt
46a9d7fc0b PA: Move in rest of purging. 2020-04-10 13:12:47 -07:00
David Goldblatt
2d6eec7b5c PA: Move in decay-all pathway. 2020-04-10 13:12:47 -07:00
David Goldblatt
65698b7f2e PA: Remove public visibility of some internals. 2020-04-10 13:12:47 -07:00
David Goldblatt
f012c43be0 PA: Move in decay_to_limit 2020-04-10 13:12:47 -07:00
David Goldblatt
103f5feda5 Move bg thread activity check out of purging core. 2020-04-10 13:12:47 -07:00
David Goldblatt
3034f4a508 PA: Move in decay_stashed. 2020-04-10 13:12:47 -07:00
David Goldblatt
aef28b2f8f PA: Move in stash_decayed. 2020-04-10 13:12:47 -07:00
David Goldblatt
655a096343 Move bg inactivity check out of purge inner loop. 
I.e. do it once per call to arena_decay_stashed instead of once per muzzy purge.
 2020-04-10 13:12:47 -07:00
David Goldblatt
71fc0dc968 PA: Move in remaining page allocation functions. 2020-04-10 13:12:47 -07:00
David Goldblatt
74958567a4 PA: have expand take sizes instead of new usize. 
This avoids involving usize, which makes some of the stats modifications more
intuitively correct.
 2020-04-10 13:12:47 -07:00
David Goldblatt
5bcc2c2ab9 PA: Have expand take szind and slab. 
This isn't really necessary, but having a uniform API will help us later.
 2020-04-10 13:12:47 -07:00
David Goldblatt
0880c2ab97 PA: Have large expands use it. 2020-04-10 13:12:47 -07:00
David Goldblatt
7be3dea82c PA: Have slab allocations use it. 2020-04-10 13:12:47 -07:00
David Goldblatt
9f93625c14 PA: Move in arena large allocation functionality. 2020-04-10 13:12:47 -07:00
David Goldblatt
7624043a41 PA: Add ehook-getting support. 2020-04-10 13:12:47 -07:00
David Goldblatt
eba35e2e48 Remove extent knowledge of arena. 2020-04-10 13:12:47 -07:00
David Goldblatt
e77f47a85a Move arena decay getters to PA. 2020-04-10 13:12:47 -07:00
David Goldblatt
f77cec311e Decay: Take current time as an argument. 
This better facilitates testing.
 2020-04-10 13:12:47 -07:00
David Goldblatt
d1d7e1076b Decay: move in some background_thread accesses. 2020-04-10 13:12:47 -07:00
David Goldblatt
8f2193dc8d Decay: Move in arena decay functions. 2020-04-10 13:12:47 -07:00
David Goldblatt
4d090d23f1 Decay: Introduce a stub .c file. 2020-04-10 13:12:47 -07:00
David Goldblatt
7b62885476 Introduce decay module and put decay objects in PA 2020-04-10 13:12:47 -07:00
David Goldblatt
3192d6b77d Extents: Have extent_dalloc_gap take ehooks. 
We're almost to the point where the extent code doesn't know about arenas at
all.  In that world, we shouldn't pull them out of the arena.
 2020-04-10 13:12:47 -07:00
David Goldblatt
22a0a7b93a Move arena_decay_extent to extent module. 2020-04-10 13:12:47 -07:00
David Goldblatt
70d12ffa05 PA: Move mapped into pa stats. 2020-04-10 13:12:47 -07:00
David Goldblatt
ce8c0d6c09 PA: Move in arena extent_sn counter. 
Just another step towards making PA self-contained.
 2020-04-10 13:12:47 -07:00
David Goldblatt
1ada4aef84 PA: Get rid of arena_ind_get calls. 
This is another step on the path towards breaking the extent reliance on the
arena module.
 2020-04-10 13:12:47 -07:00
David Goldblatt
1ad368c8b7 PA: Move in decay stats. 2020-04-10 13:12:47 -07:00
David Goldblatt
356aaa7dc6 Introduce lockedint module. 
This pulls out the various abstractions where some stats counter is sometimes an
atomic, sometimes a plain variable, sometimes always protected by a lock,
sometimes protected by reads but not writes, etc.  With this change, these cases
are treated consistently, and access patterns tagged.

In the process, we fix a few missed-update bugs (where one caller assumes
"protected-by-a-lock" semantics and another does not).
 2020-04-10 13:12:47 -07:00
David Goldblatt
acd0bf6a26 PA: move in ecache_grow. 2020-04-10 13:12:47 -07:00
David Goldblatt
32cb7c2f0b PA: Add a stats type. 2020-04-10 13:12:47 -07:00
David Goldblatt
688fb3eb89 PA: Move in the arena edata_cache. 2020-04-10 13:12:47 -07:00
David Goldblatt
8433ad84ea PA: move in shard initialization. 2020-04-10 13:12:47 -07:00
David Goldblatt
a24faed569 PA: Move in the ecache_t objects. 2020-04-10 13:12:47 -07:00
David Goldblatt
585f925055 Move cache index randomization out of extent. 
This is logically at a higher level of the stack; extent should just allocate
things at the page-level; it shouldn't care exactly why the callers wants a
given number of pages.
 2020-04-10 13:12:47 -07:00
David Goldblatt
12be9f5727 Add a stub PA module -- a page allocator. 2020-04-10 13:12:47 -07:00
Yinan Zhang
c4e9ea8cc6 Get rid of locks in prof recent test 2020-04-07 17:22:24 -07:00
Yinan Zhang
2deabac079 Get rid of custom iterator for last-N records 2020-04-07 17:22:24 -07:00
Yinan Zhang
a5ddfa7d91 Use ql for prof last-N list 2020-04-07 17:22:24 -07:00
Yinan Zhang
f9aad7a49b Add piping API to buffered writer 2020-04-01 09:41:20 -07:00
Yinan Zhang
09cd79495f Encapsulate buffer allocation failure in buffered writer 2020-04-01 09:41:20 -07:00
Yinan Zhang
a166c20818 Make prof_tctx_t pointer a true prof atomic fence 2020-03-31 17:43:42 -07:00
David T. Goldblatt
d936b46d3a Add malloc_conf_2_conf_harder 
This comes in handy when you're just a user of a canary system who wants to
change settings set by the configuration system itself.
 2020-03-31 06:25:08 -07:00
Yinan Zhang
2256ef8961 Add option to fetch system thread name on each prof sample 2020-03-24 21:39:57 -07:00
Yinan Zhang
b30a5c2f90 Reorganize cpp APIs and suppress unused function warnings 2020-03-13 12:16:09 -07:00
David Goldblatt
2e5899c129 Stats: Fix tcache_bytes reporting. 
Previously, large allocations in tcaches would have their sizes reduced during
stats estimation.  Added a test, which fails before this change but passes now.

This fixes a bug introduced in 5934846612, which
was itself fixing a bug introduced in 9c0549007d.
 2020-03-13 07:53:34 -07:00
Yinan Zhang
a5780598b3 Remove thread_event_rollback() 2020-03-12 13:55:00 -07:00
Yinan Zhang
ba783b3a0f Remove prof -> thread_event dependency 2020-03-12 13:55:00 -07:00
Yinan Zhang
441d88d1c7 Rewrite profiling thread event 2020-03-12 13:55:00 -07:00
David Goldblatt
0dcd576600 Edata cache: atomic fetch-add -> load-store. 
The modifications to count are protected by a mutex; there's no need to use the
more costly version.
 2020-03-12 11:58:09 -07:00
David Goldblatt
99b1291d17 Edata cache: add edata_cache_small_t. 
This can be used to amortize the synchronization costs of edata_cache accesses.
 2020-03-12 11:58:09 -07:00
David Goldblatt
d701a085c2 Fast path: allow low-water mark changes. 
This lets us put more allocations on an "almost as fast" path after a flush.
This results in around a 4% reduction in malloc cycles in prod workloads
(corresponding to about a 0.1% reduction in overall cycles).
 2020-03-12 11:54:19 -07:00
David Goldblatt
397da03865 Cache bin: rewrite to track more state. 
With this, we track all of the empty, full, and low water states together.  This
simplifies a lot of the tracking logic, since we now don't need the
cache_bin_info_t for state queries (except for some debugging).
 2020-03-12 11:54:19 -07:00
David Goldblatt
fef0b1ffe4 Cache bin: Remove last internals accesses. 2020-03-12 11:54:19 -07:00
David Goldblatt
0a2fcfac01 Tcache: Hold cache bin allocation explicitly. 2020-03-12 11:54:19 -07:00
David Goldblatt
d498a4bb08 Cache bin: Add an emptiness assertion. 2020-03-12 11:54:19 -07:00
David Goldblatt
6a7aa46ef7 Cache bin: Add a debug method for init checking. 2020-03-12 11:54:19 -07:00
David Goldblatt
7f5ebd211c Cache bin: set low-water internally. 2020-03-12 11:54:19 -07:00
David Goldblatt
60113dfe3b Cache bin: Move in initialization code. 2020-03-12 11:54:19 -07:00
David Goldblatt
44529da852 Cache-bin: Make flush modifications internal 
I.e. the tcache code just calls a cache-bin function to finish flush (and move
pointers around, etc.).  It doesn't directly access the cache-bin's owned memory
any more.
 2020-03-12 11:54:19 -07:00
David Goldblatt
ff6acc6ed5 Cache bin: simplify names and argument ordering. 
We always start with the cache bin, then its info (if necessary).
 2020-03-12 11:54:19 -07:00
David Goldblatt
e1dcc557d6 Cache bin: Only take the relevant cache_bin_info_t 
Previously, we took an array of cache_bin_info_ts and an index, and dereferenced
ourselves.  But infos for other cache_bins aren't relevant to any particular
cache bin, so that should be the caller's job.
 2020-03-12 11:54:19 -07:00
David Goldblatt
1b00d808d7 cache_bin: Don't let arena see empty position. 2020-03-12 11:54:19 -07:00