Commit Graph

1647 Commits

Author SHA1 Message Date
Yinan Zhang
9f71b5779b Output prof stats in stats print 2021-01-07 20:39:49 -08:00
Yinan Zhang
1f1a0231ed Split macros for initializing stats headers 2021-01-07 20:39:49 -08:00
Yinan Zhang
54f3351f1f Add mallctl for prof stats fetching 2021-01-07 20:39:49 -08:00
Yinan Zhang
40fa4d29d3 Track per size class internal fragmentation 2021-01-07 20:39:49 -08:00
Yinan Zhang
afa489c3c5 Record request size in prof info 2021-01-07 20:39:49 -08:00
David Goldblatt
f9bb8dedef Un-force-inline do_rallocx.
The additional overhead of the function-call setup and flags checking is
relatively small, but costs us the replication of the entire realloc pathway in
terms of size.
2021-01-04 14:55:49 -08:00
David Goldblatt
a9fa2defdb Add JEMALLOC_COLD, and mark some functions cold.
This hints to the compiler that it should care more about space than CPU (among
other things).  In cases where the compiler lacks profile-guided information,
this can be a substantial space savings.

For now, we mark the mallctl or atexit driven profiling and stats functions that
take up the most space.
2021-01-04 14:55:49 -08:00
David Goldblatt
5d8e70ab26 prof_recent: cassert(config_prof) more often.
This tells the compiler that these functions are never called, which lets them
be optimized away in builds where profiling is disabled.
2021-01-04 14:55:49 -08:00
David Goldblatt
83cad746ae prof_log: cassert(config_prof) in public functions
This lets the compiler infer that the code is dead in builds where profiling is
enabled, saving on space there.
2021-01-04 14:55:49 -08:00
David Goldblatt
526180b76d Extent.c: Avoid an rtree NULL-check.
The edge case in which pages_map returns (void *)PAGE can trigger an incorrect
assertion failure.  Avoid it.
2021-01-04 14:50:49 -08:00
Yinan Zhang
b35ac00d58 Do not bump to large size for page aligned request 2020-12-29 17:09:58 -08:00
Yinan Zhang
8a56d6b636 Add last-N mutex stats 2020-12-29 09:44:19 -08:00
Yinan Zhang
22d62d8cbd Handle ending gap properly for HPA stats 2020-12-18 16:40:57 -08:00
Yinan Zhang
6c5a3a24dd Omit bin stats rows with no data 2020-12-18 16:40:57 -08:00
Yinan Zhang
ea013d8fa4 Enforce realloc sizing stability 2020-12-18 11:41:52 -08:00
Yinan Zhang
74bd63b203 Optimize stats print using partial name-to-mib 2020-12-18 10:39:58 -08:00
Yinan Zhang
4557c0a67d Enable ctl on partial mib and partial name 2020-12-18 10:39:58 -08:00
Yinan Zhang
006dd0414e Add partial name-to-mib functionality 2020-12-18 10:39:58 -08:00
Yinan Zhang
f2e1a5be77 Do not fail on partial ctl path for ctl_nametomib()
We do not fail on partial ctl path when the given `mib` array is
shorter than the given name, and we should keep the behavior the
same in the reverse case, which I feel is also the more natural way.
2020-12-18 10:39:58 -08:00
Yinan Zhang
6ab181d2b7 Extract node lookup given mib input 2020-12-18 10:39:58 -08:00
Yinan Zhang
3a627b9674 No need to record all nodes in ctl_lookup() 2020-12-18 10:39:58 -08:00
Yinan Zhang
91e006c4c2 Enable ctl_lookup() to start from arbitrary node 2020-12-18 10:39:58 -08:00
Jin Qian
4e3fe218e9 Use posix_madvise to purge pages when available 2020-12-18 10:05:59 -08:00
David Goldblatt
1e3b8636ff HPA: Remove unused malloc_conf options. 2020-12-08 12:10:48 -08:00
Aditya Kumar
9522ae41d6 Move n_search outside of assert as reported by static analyzer 2020-12-07 06:49:27 -08:00
David Goldblatt
a559caf74a hpdata: Strengthen assertions.
Now that we have flat bitmap bit counting functions, we can easily assert that
nfree is always correct.  While we're tightening up this code, enforce
consistency on API boundaries as well.
2020-12-07 06:21:08 -08:00
David Goldblatt
3ed0b4e8a3 HPA: Add an nevictions counter.
I.e. the number of times we've purged a hugepage-sized region.
2020-12-07 06:21:08 -08:00
David Goldblatt
fffcefed33 malloc_conf: Clarify HPA options. 2020-12-07 06:21:08 -08:00
David Goldblatt
f7cf23aa4d psset: Relegate alloc/dalloc to test code.
This is no longer part of the "core" functionality; we only need the stub
implementations as an end-to-end test of hpdata + psset interactions when
metadata is being modified.  Treat them accordingly.
2020-12-07 06:21:08 -08:00
David Goldblatt
f9299ca572 HPA: Use psset fit/insert/remove.
This will let us remove alloc_new and alloc_reuse functions from the psset.
2020-12-07 06:21:08 -08:00
David Goldblatt
0971e1e4e3 hpdata: Use addr/size instead of begin/npages.
This is easier for the users of the hpdata.
2020-12-07 06:21:08 -08:00
David Goldblatt
5228d869ee psset: Use fit/insert/remove as basis functions.
All other functionality can be implemented in terms of these; doing so (while
retaining the same API) will be convenient for subsequent refactors.
2020-12-07 06:21:08 -08:00
David Goldblatt
089f8fa442 Move hpdata bitmap logic out of the psset. 2020-12-07 06:21:08 -08:00
David Goldblatt
ca30b5db2b Introduce hpdata_t.
Using an edata_t both for hugepages and the allocations within those hugepages
was convenient at first, but has outlived its usefulness.  Representing
hugepages explicitly, with their own data structure, will make future
development easier.
2020-12-07 06:21:08 -08:00
David Goldblatt
43af63fff4 HPA: Manage whole hugepages at a time.
This redesigns the HPA implementation to allow us to manage hugepages all at
once, locally, without relying on a global fallback.
2020-12-07 06:21:08 -08:00
David Goldblatt
c1b2a77933 psset: Move in stats.
A later change will benefit from having these functions pulled into a
psset-module set of functions.
2020-12-07 06:21:08 -08:00
David Goldblatt
d0a991d47b psset: Add insert/remove functions.
These will allow us to (for instance) move pageslabs from a psset dedicated to
not-yet-hugeified pages to one dedicated to hugeified ones.
2020-12-07 06:21:08 -08:00
David Goldblatt
d438296b1f narenas_ratio: Accept fractional values.
With recent scalability improvements to the HPA, we're experimenting with much
lower arena counts; this gets annoying when trying to test across different
hardware configurations using only the narenas setting.
2020-12-04 23:48:19 -08:00
David Goldblatt
ecd39418ac Add fxp: A fixed-point math library.
This will be used in the next commit to allow non-integer values for
narenas_ratio.
2020-12-04 23:48:19 -08:00
David Carlier
520b75fa2d utrace support with label based signature. 2020-11-30 11:43:00 -08:00
Yinan Zhang
92e189be8b Add some comments to the batch allocation logic flow 2020-11-16 20:58:01 -08:00
Yinan Zhang
d96e4525ad Route batch allocation of small batch size to tcache 2020-11-16 20:58:01 -08:00
Yinan Zhang
566c4a8594 Slight changes to cache bin internal functions 2020-11-16 20:58:01 -08:00
Yinan Zhang
9545c2cd36 Add sample interval to prof last-N dump 2020-11-13 15:33:27 -08:00
David Goldblatt
cf2549a149 Add a per-arena oversize_threshold.
This can let manual arenas trade off memory and CPU the way auto arenas do.
2020-11-13 13:45:35 -08:00
David Goldblatt
4ca3d91e96 Rename geom_grow -> exp_grow.
This was promised in the review of the introduction of geom_grow, but would have
been painful to do there because of the series that introduced it.  Now that
those are comitted, renaming is easier.
2020-11-13 13:42:33 -08:00
David Goldblatt
b4c37a6e81 Rename edata_tree_t -> edata_avail_t.
This isn't a tree any more, and it mildly irritates me any time I see it.
2020-11-13 13:42:11 -08:00
David Carlier
95f0a77fde Detect pthread_getname_np explicitly.
At least one libc (musl) defines pthread_setname_np without defining
pthread_getname_np. Detect the presence of each individually, rather than
inferring both must be defined if set is.
2020-11-11 17:31:22 -08:00
David Goldblatt
589638182a Use the edata_cache_small_t in the HPA. 2020-11-05 12:34:43 -08:00
David Goldblatt
03a6047111 Edata cache small: rewrite.
In previous designs, this was intended to be a sort of cache that couldn't fail.
In the current design, we want to use it just as a contention reduction
mechanism.  Rewrite it with those goals in mind.
2020-11-05 12:34:43 -08:00