project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
36c6bfb963
server-skynet-source-3rd-je.../include/jemalloc/internal/psset.h

125 lines
3.9 KiB
C
Raw Normal View History

Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
#ifndef JEMALLOC_INTERNAL_PSSET_H
#define JEMALLOC_INTERNAL_PSSET_H
Introduce hpdata_t. Using an edata_t both for hugepages and the allocations within those hugepages was convenient at first, but has outlived its usefulness. Representing hugepages explicitly, with their own data structure, will make future development easier.
2020-11-18 08:32:45 +08:00
#include "jemalloc/internal/hpdata.h"
Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
/*
* A page-slab set. What the eset is to PAC, the psset is to HPA. It maintains
* a collection of page-slabs (the intent being that they are backed by
* hugepages, or at least could be), and handles allocation and deallocation
* requests.
*/
/*
* One more than the maximum pszind_t we will serve out of the HPA.
* Practically, we expect only the first few to be actually used. This
* corresponds to a maximum size of of 512MB on systems with 4k pages and
* SC_NGROUP == 4, which is already an unreasonably large maximum. Morally, you
* can think of this as being SC_NPSIZES, but there's no sense in wasting that
* much space in the arena, making bitmaps that much larger, etc.
*/
#define PSSET_NPSIZES 64
HPA: Use dirtiest-first purging. This seems to be practically beneficial, despite some pathological corner cases.
2021-02-09 06:11:37 +08:00
/*
* We keep two purge lists per page size class; one for hugified hpdatas (at
* index 2*pszind), and one for the non-hugified hpdatas (at index 2*pszind +
* 1). This lets us implement a preference for purging non-hugified hpdatas
* among similarly-dirty ones.
*/
#define PSSET_NPURGE_LISTS (2 * PSSET_NPSIZES)
psset: Add stats
2020-09-03 03:59:10 +08:00
typedef struct psset_bin_stats_s psset_bin_stats_t;
struct psset_bin_stats_s {
/* How many pageslabs are in this bin? */
psset stats: Simplify handling. We can treat the huge and nonhuge cases uniformly using huge state as an array index.
2020-12-04 10:32:42 +08:00
size_t npageslabs;
psset: Add stats
2020-09-03 03:59:10 +08:00
/* Of them, how many pages are active? */
psset stats: Simplify handling. We can treat the huge and nonhuge cases uniformly using huge state as an array index.
2020-12-04 10:32:42 +08:00
size_t nactive;
HPA: Track dirty stats.
2020-12-04 11:15:54 +08:00
/* And how many are dirty? */
size_t ndirty;
psset: Add stats
2020-09-03 03:59:10 +08:00
};
psset: Move in stats. A later change will benefit from having these functions pulled into a psset-module set of functions.
2020-11-11 08:23:03 +08:00
typedef struct psset_stats_s psset_stats_t;
struct psset_stats_s {
psset stats: Simplify handling. We can treat the huge and nonhuge cases uniformly using huge state as an array index.
2020-12-04 10:32:42 +08:00
/*
* The second index is huge stats; nonfull_slabs[pszind][0] contains
* stats for the non-huge slabs in bucket pszind, while
* nonfull_slabs[pszind][1] contains stats for the huge slabs.
*/
psset_bin_stats_t nonfull_slabs[PSSET_NPSIZES][2];
psset: Move in stats. A later change will benefit from having these functions pulled into a psset-module set of functions.
2020-11-11 08:23:03 +08:00
/*
Move empty slab tracking to the psset. We're moving towards a world in which purging decisions are less rigidly enforced at a single-hugepage level. In that world, it makes sense to keep around some hpdatas which are not completely purged, in which case we'll need to track them.
2020-12-06 09:42:04 +08:00
* Full slabs don't live in any edata heap, but we still track their
psset: Move in stats. A later change will benefit from having these functions pulled into a psset-module set of functions.
2020-11-11 08:23:03 +08:00
* stats.
*/
psset stats: Simplify handling. We can treat the huge and nonhuge cases uniformly using huge state as an array index.
2020-12-04 10:32:42 +08:00
psset_bin_stats_t full_slabs[2];
Move empty slab tracking to the psset. We're moving towards a world in which purging decisions are less rigidly enforced at a single-hugepage level. In that world, it makes sense to keep around some hpdatas which are not completely purged, in which case we'll need to track them.
2020-12-06 09:42:04 +08:00
/* Empty slabs are similar. */
psset_bin_stats_t empty_slabs[2];
psset: Move in stats. A later change will benefit from having these functions pulled into a psset-module set of functions.
2020-11-11 08:23:03 +08:00
};
HPA: Add stats for the hpa_shard.
2020-09-05 03:01:52 +08:00
Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
typedef struct psset_s psset_t;
struct psset_s {
/*
* The pageslabs, quantized by the size class of the largest contiguous
* free run of pages in a pageslab.
*/
Introduce hpdata_t. Using an edata_t both for hugepages and the allocations within those hugepages was convenient at first, but has outlived its usefulness. Representing hugepages explicitly, with their own data structure, will make future development easier.
2020-11-18 08:32:45 +08:00
hpdata_age_heap_t pageslabs[PSSET_NPSIZES];
Use the flat bitmap for eset and psset bitmaps. This is simpler (note that the eset field comment was actually incorrect!), and slightly faster.
2021-02-09 03:04:46 +08:00
/* Bitmap for which set bits correspond to non-empty heaps. */
psset: Rename "bitmap" to "pageslab_bitmap". It tracks pageslabs. Soon, we'll have another bitmap (to track dirty pages) that we want to disambiguate. While we're here, fix an out-of-date comment.
2021-02-09 03:26:56 +08:00
fb_group_t pageslab_bitmap[FB_NGROUPS(PSSET_NPSIZES)];
psset: keep aggregate stats. This will let us quickly query these stats to make purging decisions quickly.
2020-12-07 04:49:03 +08:00
/*
* The sum of all bin stats in stats. This lets us quickly answer
* queries for the number of dirty, active, and retained pages in the
* entire set.
*/
psset_bin_stats_t merged_stats;
psset: Move in stats. A later change will benefit from having these functions pulled into a psset-module set of functions.
2020-11-11 08:23:03 +08:00
psset_stats_t stats;
HPA: Track pending purges/hugifies in the psset. This finishes the refactoring of the HPA/psset interactions the past few commits have been building towards. Rather than the HPA removing and then reinserting hpdatas, it simply begins updates and ends them. These updates can set flags on the hpdata that prevent it from being returned for certain types of requests. For example, it can call hpdata_alloc_allowed_set(hpdata, false) during an update, at which point the given hpdata will no longer be returned for psset_pick_alloc requests. This has various of benefits: - It maintains stats correctness during purges and hugifies. - It allows simpler and more explicit concurrency control for the various special cases (e.g. allocations are disallowed during purge, but not during hugify). - It lets allocations and deallocations avoid disturbing the purging and hugification orderings. If an hpdata "loses its place" in one of the queues just do to an alloc / dalloc, it can result in pathological edge cases where very hot, very full hugepages never get hugified (and cold extents on the same hugepage as hot ones never get purged). The key benefit though is that tracking hpdatas to be purged / hugified in a principled way will let us do delayed purging and hugification. Eventually this will let us move these operations to background threads, but in the short term the benefit is that it will let us have global purging policies (e.g. purge when the entire arena has too many dirty pages, rather than any particular hugepage).
2020-12-07 01:49:26 +08:00
/*
* Slabs with no active allocations, but which are allowed to serve new
* allocations.
*/
hpdata_empty_list_t empty;
HPA: Make purging/hugifying more principled. Before this change, purge/hugify decisions had several sharp edges that could lead to pathological behavior if tuning parameters weren't carefully chosen. It's the first of a series; this introduces basic "make every hugepage with dirty pages purgeable" functionality, and the next commit expands that functionality to have a smarter policy for picking hugepages to purge. Previously, the dehugify logic would *never* dehugify a hugepage unless it was dirtier than the dehugification threshold. This can lead to situations in which these pages (which themselves could never be purged) would push us above the maximum allowed dirty pages in the shard. This forces immediate purging of any pages deallocated in non-hugified hugepages, which in turn places nonobvious practical limitations on the relationships between various config settings. Instead, we make our preference not to dehugify to purge a soft one rather than a hard one. We'll avoid purging them, but only so long as we can do so by purging non-hugified pages. If we need to purge them to satisfy our dirty page limits, or to hugify other, more worthy candidates, we'll still do so.
2021-02-06 02:46:17 +08:00
/* Slabs which are available to be purged, ordered by purge level. */
HPA: Use dirtiest-first purging. This seems to be practically beneficial, despite some pathological corner cases.
2021-02-09 06:11:37 +08:00
hpdata_purge_list_t to_purge[PSSET_NPURGE_LISTS];
/* Bitmap for which set bits correspond to non-empty purge lists. */
fb_group_t purge_bitmap[FB_NGROUPS(PSSET_NPURGE_LISTS)];
HPA: Track pending purges/hugifies in the psset. This finishes the refactoring of the HPA/psset interactions the past few commits have been building towards. Rather than the HPA removing and then reinserting hpdatas, it simply begins updates and ends them. These updates can set flags on the hpdata that prevent it from being returned for certain types of requests. For example, it can call hpdata_alloc_allowed_set(hpdata, false) during an update, at which point the given hpdata will no longer be returned for psset_pick_alloc requests. This has various of benefits: - It maintains stats correctness during purges and hugifies. - It allows simpler and more explicit concurrency control for the various special cases (e.g. allocations are disallowed during purge, but not during hugify). - It lets allocations and deallocations avoid disturbing the purging and hugification orderings. If an hpdata "loses its place" in one of the queues just do to an alloc / dalloc, it can result in pathological edge cases where very hot, very full hugepages never get hugified (and cold extents on the same hugepage as hot ones never get purged). The key benefit though is that tracking hpdatas to be purged / hugified in a principled way will let us do delayed purging and hugification. Eventually this will let us move these operations to background threads, but in the short term the benefit is that it will let us have global purging policies (e.g. purge when the entire arena has too many dirty pages, rather than any particular hugepage).
2020-12-07 01:49:26 +08:00
/* Slabs which are available to be hugified. */
hpdata_hugify_list_t to_hugify;
Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
};
void psset_init(psset_t *psset);
psset: Move in stats. A later change will benefit from having these functions pulled into a psset-module set of functions.
2020-11-11 08:23:03 +08:00
void psset_stats_accum(psset_stats_t *dst, psset_stats_t *src);
Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
psset: Reconceptualize insertion/removal. Really, this isn't a functional change, just a naming change. We start thinking of pageslabs as being always in the psset. What we used to think of as removal is now thought of as being in the psset, but in the process of being updated (and therefore, unavalable for serving new allocations). This is in preparation of subsequent changes to support deferred purging; allocations will still be in the psset for the purposes of choosing when to purge, but not for purposes of allocation/deallocation.
2020-12-06 07:58:31 +08:00
/*
* Begin or end updating the given pageslab's metadata. While the pageslab is
* being updated, it won't be returned from psset_fit calls.
*/
void psset_update_begin(psset_t *psset, hpdata_t *ps);
void psset_update_end(psset_t *psset, hpdata_t *ps);
psset: Add insert/remove functions. These will allow us to (for instance) move pageslabs from a psset dedicated to not-yet-hugeified pages to one dedicated to hugeified ones.
2020-11-10 09:24:31 +08:00
psset: Use fit/insert/remove as basis functions. All other functionality can be implemented in terms of these; doing so (while retaining the same API) will be convenient for subsequent refactors.
2020-12-01 05:28:54 +08:00
/* Analogous to the eset_fit; pick a hpdata to serve the request. */
psset: Reconceptualize insertion/removal. Really, this isn't a functional change, just a naming change. We start thinking of pageslabs as being always in the psset. What we used to think of as removal is now thought of as being in the psset, but in the process of being updated (and therefore, unavalable for serving new allocations). This is in preparation of subsequent changes to support deferred purging; allocations will still be in the psset for the purposes of choosing when to purge, but not for purposes of allocation/deallocation.
2020-12-06 07:58:31 +08:00
hpdata_t *psset_pick_alloc(psset_t *psset, size_t size);
HPA: Track pending purges/hugifies in the psset. This finishes the refactoring of the HPA/psset interactions the past few commits have been building towards. Rather than the HPA removing and then reinserting hpdatas, it simply begins updates and ends them. These updates can set flags on the hpdata that prevent it from being returned for certain types of requests. For example, it can call hpdata_alloc_allowed_set(hpdata, false) during an update, at which point the given hpdata will no longer be returned for psset_pick_alloc requests. This has various of benefits: - It maintains stats correctness during purges and hugifies. - It allows simpler and more explicit concurrency control for the various special cases (e.g. allocations are disallowed during purge, but not during hugify). - It lets allocations and deallocations avoid disturbing the purging and hugification orderings. If an hpdata "loses its place" in one of the queues just do to an alloc / dalloc, it can result in pathological edge cases where very hot, very full hugepages never get hugified (and cold extents on the same hugepage as hot ones never get purged). The key benefit though is that tracking hpdatas to be purged / hugified in a principled way will let us do delayed purging and hugification. Eventually this will let us move these operations to background threads, but in the short term the benefit is that it will let us have global purging policies (e.g. purge when the entire arena has too many dirty pages, rather than any particular hugepage).
2020-12-07 01:49:26 +08:00
/* Pick one to purge. */
hpdata_t *psset_pick_purge(psset_t *psset);
/* Pick one to hugify. */
hpdata_t *psset_pick_hugify(psset_t *psset);
Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
Move empty slab tracking to the psset. We're moving towards a world in which purging decisions are less rigidly enforced at a single-hugepage level. In that world, it makes sense to keep around some hpdatas which are not completely purged, in which case we'll need to track them.
2020-12-06 09:42:04 +08:00
void psset_insert(psset_t *psset, hpdata_t *ps);
void psset_remove(psset_t *psset, hpdata_t *ps);
psset: keep aggregate stats. This will let us quickly query these stats to make purging decisions quickly.
2020-12-07 04:49:03 +08:00
static inline size_t
psset_npageslabs(psset_t *psset) {
return psset->merged_stats.npageslabs;
}
static inline size_t
psset_nactive(psset_t *psset) {
return psset->merged_stats.nactive;
}
static inline size_t
psset_ndirty(psset_t *psset) {
return psset->merged_stats.ndirty;
}
Add psset: a set of pageslabs. This introduces a new sort of edata_t; a pageslab, and a set to manage them. This is part of a series of a commits to implement a hugepage allocator; the pageset will be per-arena, and track small page allocations requests within a larger extent allocated from a centralized hugepage allocator.
2020-07-11 08:40:13 +08:00
#endif /* JEMALLOC_INTERNAL_PSSET_H */
Reference in New Issue Copy Permalink