project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
162c2bcf31
server-skynet-source-3rd-je.../test/unit/bitmap.c

434 lines
11 KiB
C
Raw Normal View History

Refactor to support more varied testing. Refactor the test harness to support three types of tests: - unit: White box unit tests. These tests have full access to all internal jemalloc library symbols. Though in actuality all symbols are prefixed by jet_, macro-based name mangling abstracts this away from test code. - integration: Black box integration tests. These tests link with the installable shared jemalloc library, and with the exception of some utility code and configure-generated macro definitions, they have no access to jemalloc internals. - stress: Black box stress tests. These tests link with the installable shared jemalloc library, as well as with an internal allocator with symbols prefixed by jet_ (same as for unit tests) that can be used to allocate data structures that are internal to the test code. Move existing tests into test/{unit,integration}/ as appropriate. Split out internal parts of jemalloc_defs.h.in and put them in jemalloc_internal_defs.h.in. This reduces internals exposure to applications that #include <jemalloc/jemalloc.h>. Refactor jemalloc.h header generation so that a single header file results, and the prototypes can be used to generate jet_ prototypes for tests. Split jemalloc.h.in into multiple parts (jemalloc_defs.h.in, jemalloc_macros.h.in, jemalloc_protos.h.in, jemalloc_mangle.h.in) and use a shell script to generate a unified jemalloc.h at configure time. Change the default private namespace prefix from "" to "je_". Add missing private namespace mangling. Remove hard-coded private_namespace.h. Instead generate it and private_unnamespace.h from private_symbols.txt. Use similar logic for public symbols, which aids in name mangling for jet_ symbols. Add test_warn() and test_fail(). Replace existing exit(1) calls with test_fail() calls.
2013-12-01 07:25:42 +08:00
#include "test/jemalloc_test.h"
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NBITS_TAB \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
NB( 1) \
NB( 2) \
NB( 3) \
NB( 4) \
NB( 5) \
NB( 6) \
NB( 7) \
NB( 8) \
NB( 9) \
NB(10) \
NB(11) \
NB(12) \
NB(13) \
NB(14) \
NB(15) \
NB(16) \
NB(17) \
NB(18) \
NB(19) \
NB(20) \
NB(21) \
NB(22) \
NB(23) \
NB(24) \
NB(25) \
NB(26) \
NB(27) \
NB(28) \
NB(29) \
NB(30) \
NB(31) \
NB(32) \
\
NB(33) \
NB(34) \
NB(35) \
NB(36) \
NB(37) \
NB(38) \
NB(39) \
NB(40) \
NB(41) \
NB(42) \
NB(43) \
NB(44) \
NB(45) \
NB(46) \
NB(47) \
NB(48) \
NB(49) \
NB(50) \
NB(51) \
NB(52) \
NB(53) \
NB(54) \
NB(55) \
NB(56) \
NB(57) \
NB(58) \
NB(59) \
NB(60) \
NB(61) \
NB(62) \
NB(63) \
NB(64) \
NB(65) \
\
NB(126) \
NB(127) \
NB(128) \
NB(129) \
NB(130) \
\
NB(254) \
NB(255) \
NB(256) \
NB(257) \
NB(258) \
\
NB(510) \
NB(511) \
NB(512) \
NB(513) \
NB(514) \
\
NB(1024) \
NB(2048) \
NB(4096) \
NB(8192) \
NB(16384) \
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
test_bitmap_initializer_body(const bitmap_info_t *binfo, size_t nbits) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_t binfo_dyn;
bitmap_info_init(&binfo_dyn, nbits);
assert_zu_eq(bitmap_size(binfo), bitmap_size(&binfo_dyn),
"Unexpected difference between static and dynamic initialization, "
"nbits=%zu", nbits);
assert_zu_eq(binfo->nbits, binfo_dyn.nbits,
"Unexpected difference between static and dynamic initialization, "
"nbits=%zu", nbits);
Revert "Remove BITMAP_USE_TREE." Some systems use a native 64 KiB page size, which means that the bitmap for the smallest size class can be 8192 bits, not just 512 bits as when the page size is 4 KiB. Linear search in bitmap_{sfu,ffu}() is unacceptably slow for such large bitmaps. This reverts commit 7c00f04ff40a34627e31488d02ff1081c749c7ba.
2017-04-17 00:25:56 +08:00
#ifdef BITMAP_USE_TREE
assert_u_eq(binfo->nlevels, binfo_dyn.nlevels,
"Unexpected difference between static and dynamic initialization, "
"nbits=%zu", nbits);
{
unsigned i;
for (i = 0; i < binfo->nlevels; i++) {
assert_zu_eq(binfo->levels[i].group_offset,
binfo_dyn.levels[i].group_offset,
"Unexpected difference between static and dynamic "
"initialization, nbits=%zu, level=%u", nbits, i);
}
}
#else
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
assert_zu_eq(binfo->ngroups, binfo_dyn.ngroups,
"Unexpected difference between static and dynamic initialization");
Revert "Remove BITMAP_USE_TREE." Some systems use a native 64 KiB page size, which means that the bitmap for the smallest size class can be 8192 bits, not just 512 bits as when the page size is 4 KiB. Linear search in bitmap_{sfu,ffu}() is unacceptably slow for such large bitmaps. This reverts commit 7c00f04ff40a34627e31488d02ff1081c749c7ba.
2017-04-17 00:25:56 +08:00
#endif
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_bitmap_initializer) {
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NB(nbits) { \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
if (nbits <= BITMAP_MAXBITS) { \
bitmap_info_t binfo = \
BITMAP_INFO_INITIALIZER(nbits); \
test_bitmap_initializer_body(&binfo, nbits); \
} \
}
NBITS_TAB
#undef NB
}
TEST_END
static size_t
test_bitmap_size_body(const bitmap_info_t *binfo, size_t nbits,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
size_t prev_size) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
size_t size = bitmap_size(binfo);
assert_zu_ge(size, (nbits >> 3),
"Bitmap size is smaller than expected");
assert_zu_ge(size, prev_size, "Bitmap size is smaller than expected");
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return size;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_bitmap_size) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
size_t nbits, prev_size;
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
prev_size = 0;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
for (nbits = 1; nbits <= BITMAP_MAXBITS; nbits++) {
Miscellaneous bitmap refactoring.
2016-02-27 05:59:41 +08:00
bitmap_info_t binfo;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_init(&binfo, nbits);
prev_size = test_bitmap_size_body(&binfo, nbits, prev_size);
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NB(nbits) { \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_t binfo = BITMAP_INFO_INITIALIZER(nbits); \
prev_size = test_bitmap_size_body(&binfo, nbits, \
prev_size); \
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
prev_size = 0;
NBITS_TAB
#undef NB
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Refactor tests. Refactor tests to use explicit testing assertions, rather than diff'ing test output. This makes the test code a bit shorter, more explicitly encodes testing intent, and makes test failure diagnosis more straightforward.
2013-12-09 12:52:21 +08:00
TEST_END
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
test_bitmap_init_body(const bitmap_info_t *binfo, size_t nbits) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
size_t i;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_t *bitmap = (bitmap_t *)malloc(bitmap_size(binfo));
assert_ptr_not_null(bitmap, "Unexpected malloc() failure");
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
bitmap_init(bitmap, binfo, false);
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
for (i = 0; i < nbits; i++) {
assert_false(bitmap_get(bitmap, binfo, i),
"Bit should be unset");
}
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
bitmap_init(bitmap, binfo, true);
for (i = 0; i < nbits; i++) {
assert_true(bitmap_get(bitmap, binfo, i), "Bit should be set");
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
free(bitmap);
}
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_bitmap_init) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
size_t nbits;
for (nbits = 1; nbits <= BITMAP_MAXBITS; nbits++) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
bitmap_info_t binfo;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_init(&binfo, nbits);
test_bitmap_init_body(&binfo, nbits);
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NB(nbits) { \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_t binfo = BITMAP_INFO_INITIALIZER(nbits); \
test_bitmap_init_body(&binfo, nbits); \
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
NBITS_TAB
#undef NB
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Refactor tests. Refactor tests to use explicit testing assertions, rather than diff'ing test output. This makes the test code a bit shorter, more explicitly encodes testing intent, and makes test failure diagnosis more straightforward.
2013-12-09 12:52:21 +08:00
TEST_END
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
test_bitmap_set_body(const bitmap_info_t *binfo, size_t nbits) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
size_t i;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_t *bitmap = (bitmap_t *)malloc(bitmap_size(binfo));
assert_ptr_not_null(bitmap, "Unexpected malloc() failure");
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
bitmap_init(bitmap, binfo, false);
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
for (i = 0; i < nbits; i++) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_set(bitmap, binfo, i);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
assert_true(bitmap_full(bitmap, binfo), "All bits should be set");
free(bitmap);
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_bitmap_set) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
size_t nbits;
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
for (nbits = 1; nbits <= BITMAP_MAXBITS; nbits++) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
bitmap_info_t binfo;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_init(&binfo, nbits);
test_bitmap_set_body(&binfo, nbits);
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NB(nbits) { \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_t binfo = BITMAP_INFO_INITIALIZER(nbits); \
test_bitmap_set_body(&binfo, nbits); \
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
NBITS_TAB
#undef NB
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Refactor tests. Refactor tests to use explicit testing assertions, rather than diff'ing test output. This makes the test code a bit shorter, more explicitly encodes testing intent, and makes test failure diagnosis more straightforward.
2013-12-09 12:52:21 +08:00
TEST_END
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
test_bitmap_unset_body(const bitmap_info_t *binfo, size_t nbits) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
size_t i;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_t *bitmap = (bitmap_t *)malloc(bitmap_size(binfo));
assert_ptr_not_null(bitmap, "Unexpected malloc() failure");
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
bitmap_init(bitmap, binfo, false);
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
for (i = 0; i < nbits; i++) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_set(bitmap, binfo, i);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
assert_true(bitmap_full(bitmap, binfo), "All bits should be set");
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
for (i = 0; i < nbits; i++) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_unset(bitmap, binfo, i);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
for (i = 0; i < nbits; i++) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_set(bitmap, binfo, i);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
assert_true(bitmap_full(bitmap, binfo), "All bits should be set");
free(bitmap);
}
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_bitmap_unset) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
size_t nbits;
for (nbits = 1; nbits <= BITMAP_MAXBITS; nbits++) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
bitmap_info_t binfo;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_init(&binfo, nbits);
test_bitmap_unset_body(&binfo, nbits);
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NB(nbits) { \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_t binfo = BITMAP_INFO_INITIALIZER(nbits); \
test_bitmap_unset_body(&binfo, nbits); \
}
NBITS_TAB
#undef NB
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Refactor tests. Refactor tests to use explicit testing assertions, rather than diff'ing test output. This makes the test code a bit shorter, more explicitly encodes testing intent, and makes test failure diagnosis more straightforward.
2013-12-09 12:52:21 +08:00
TEST_END
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
static void
Fix BITMAP_USE_TREE version of bitmap_ffu(). This fixes an extent searching regression on 32-bit systems, caused by the initial bitmap_ffu() implementation in c8021d01f6efe14dc1bd200021a815638063cb5f (Implement bitmap_ffu(), which finds the first unset bit.), as first used in 5d33233a5e6601902df7cddd8cc8aa0b135c77b2 (Use a bitmap in extents_t to speed up search.).
2017-03-25 13:46:56 +08:00
test_bitmap_xfu_body(const bitmap_info_t *binfo, size_t nbits) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_t *bitmap = (bitmap_t *)malloc(bitmap_size(binfo));
assert_ptr_not_null(bitmap, "Unexpected malloc() failure");
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
bitmap_init(bitmap, binfo, false);
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
/* Iteratively set bits starting at the beginning. */
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
for (size_t i = 0; i < nbits; i++) {
assert_zu_eq(bitmap_ffu(bitmap, binfo, 0), i,
"First unset bit should be just after previous first unset "
"bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, (i > 0) ? i-1 : i), i,
"First unset bit should be just after previous first unset "
"bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, i), i,
"First unset bit should be just after previous first unset "
"bit");
assert_zu_eq(bitmap_sfu(bitmap, binfo), i,
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
"First unset bit should be just after previous first unset "
"bit");
}
assert_true(bitmap_full(bitmap, binfo), "All bits should be set");
/*
* Iteratively unset bits starting at the end, and verify that
* bitmap_sfu() reaches the unset bits.
*/
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
for (size_t i = nbits - 1; i < nbits; i--) { /* (nbits..0] */
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_unset(bitmap, binfo, i);
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
assert_zu_eq(bitmap_ffu(bitmap, binfo, 0), i,
"First unset bit should the bit previously unset");
assert_zu_eq(bitmap_ffu(bitmap, binfo, (i > 0) ? i-1 : i), i,
"First unset bit should the bit previously unset");
assert_zu_eq(bitmap_ffu(bitmap, binfo, i), i,
"First unset bit should the bit previously unset");
assert_zu_eq(bitmap_sfu(bitmap, binfo), i,
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
"First unset bit should the bit previously unset");
bitmap_unset(bitmap, binfo, i);
}
assert_false(bitmap_get(bitmap, binfo, 0), "Bit should be unset");
/*
* Iteratively set bits starting at the beginning, and verify that
* bitmap_sfu() looks past them.
*/
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
for (size_t i = 1; i < nbits; i++) {
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_set(bitmap, binfo, i - 1);
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
assert_zu_eq(bitmap_ffu(bitmap, binfo, 0), i,
"First unset bit should be just after the bit previously "
"set");
assert_zu_eq(bitmap_ffu(bitmap, binfo, (i > 0) ? i-1 : i), i,
"First unset bit should be just after the bit previously "
"set");
assert_zu_eq(bitmap_ffu(bitmap, binfo, i), i,
"First unset bit should be just after the bit previously "
"set");
assert_zu_eq(bitmap_sfu(bitmap, binfo), i,
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
"First unset bit should be just after the bit previously "
"set");
bitmap_unset(bitmap, binfo, i);
}
Implement bitmap_ffu(), which finds the first unset bit.
2017-03-24 08:59:47 +08:00
assert_zu_eq(bitmap_ffu(bitmap, binfo, 0), nbits - 1,
"First unset bit should be the last bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, (nbits > 1) ? nbits-2 : nbits-1),
nbits - 1, "First unset bit should be the last bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, nbits - 1), nbits - 1,
"First unset bit should be the last bit");
assert_zu_eq(bitmap_sfu(bitmap, binfo), nbits - 1,
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
"First unset bit should be the last bit");
assert_true(bitmap_full(bitmap, binfo), "All bits should be set");
Fix BITMAP_USE_TREE version of bitmap_ffu(). This fixes an extent searching regression on 32-bit systems, caused by the initial bitmap_ffu() implementation in c8021d01f6efe14dc1bd200021a815638063cb5f (Implement bitmap_ffu(), which finds the first unset bit.), as first used in 5d33233a5e6601902df7cddd8cc8aa0b135c77b2 (Use a bitmap in extents_t to speed up search.).
2017-03-25 13:46:56 +08:00
/*
* Bubble a "usu" pattern through the bitmap and verify that
* bitmap_ffu() finds the correct bit for all five min_bit cases.
*/
if (nbits >= 3) {
for (size_t i = 0; i < nbits-2; i++) {
bitmap_unset(bitmap, binfo, i);
bitmap_unset(bitmap, binfo, i+2);
if (i > 0) {
assert_zu_eq(bitmap_ffu(bitmap, binfo, i-1), i,
"Unexpected first unset bit");
}
assert_zu_eq(bitmap_ffu(bitmap, binfo, i), i,
"Unexpected first unset bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, i+1), i+2,
"Unexpected first unset bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, i+2), i+2,
"Unexpected first unset bit");
if (i + 3 < nbits) {
assert_zu_eq(bitmap_ffu(bitmap, binfo, i+3),
nbits, "Unexpected first unset bit");
}
assert_zu_eq(bitmap_sfu(bitmap, binfo), i,
"Unexpected first unset bit");
assert_zu_eq(bitmap_sfu(bitmap, binfo), i+2,
"Unexpected first unset bit");
}
}
Fix bitmap_ffu() to work with 3+ levels.
2017-03-27 16:52:20 +08:00
/*
* Unset the last bit, bubble another unset bit through the bitmap, and
* verify that bitmap_ffu() finds the correct bit for all four min_bit
* cases.
*/
if (nbits >= 3) {
bitmap_unset(bitmap, binfo, nbits-1);
for (size_t i = 0; i < nbits-1; i++) {
bitmap_unset(bitmap, binfo, i);
if (i > 0) {
assert_zu_eq(bitmap_ffu(bitmap, binfo, i-1), i,
"Unexpected first unset bit");
}
assert_zu_eq(bitmap_ffu(bitmap, binfo, i), i,
"Unexpected first unset bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, i+1), nbits-1,
"Unexpected first unset bit");
assert_zu_eq(bitmap_ffu(bitmap, binfo, nbits-1),
nbits-1, "Unexpected first unset bit");
assert_zu_eq(bitmap_sfu(bitmap, binfo), i,
"Unexpected first unset bit");
}
assert_zu_eq(bitmap_sfu(bitmap, binfo), nbits-1,
"Unexpected first unset bit");
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
free(bitmap);
}
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
Fix BITMAP_USE_TREE version of bitmap_ffu(). This fixes an extent searching regression on 32-bit systems, caused by the initial bitmap_ffu() implementation in c8021d01f6efe14dc1bd200021a815638063cb5f (Implement bitmap_ffu(), which finds the first unset bit.), as first used in 5d33233a5e6601902df7cddd8cc8aa0b135c77b2 (Use a bitmap in extents_t to speed up search.).
2017-03-25 13:46:56 +08:00
TEST_BEGIN(test_bitmap_xfu) {
Limit # of iters of test_bitmap_xfu. Otherwise the test is too slow for higher page sizes such as 64k.
2019-09-12 01:21:46 +08:00
size_t nbits, nbits_max;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
Limit # of iters of test_bitmap_xfu. Otherwise the test is too slow for higher page sizes such as 64k.
2019-09-12 01:21:46 +08:00
/* The test is O(n^2); large page sizes may slow down too much. */
nbits_max = BITMAP_MAXBITS > 512 ? 512 : BITMAP_MAXBITS;
for (nbits = 1; nbits <= nbits_max; nbits++) {
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
bitmap_info_t binfo;
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_init(&binfo, nbits);
Fix BITMAP_USE_TREE version of bitmap_ffu(). This fixes an extent searching regression on 32-bit systems, caused by the initial bitmap_ffu() implementation in c8021d01f6efe14dc1bd200021a815638063cb5f (Implement bitmap_ffu(), which finds the first unset bit.), as first used in 5d33233a5e6601902df7cddd8cc8aa0b135c77b2 (Use a bitmap in extents_t to speed up search.).
2017-03-25 13:46:56 +08:00
test_bitmap_xfu_body(&binfo, nbits);
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
}
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NB(nbits) { \
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
bitmap_info_t binfo = BITMAP_INFO_INITIALIZER(nbits); \
Fix BITMAP_USE_TREE version of bitmap_ffu(). This fixes an extent searching regression on 32-bit systems, caused by the initial bitmap_ffu() implementation in c8021d01f6efe14dc1bd200021a815638063cb5f (Implement bitmap_ffu(), which finds the first unset bit.), as first used in 5d33233a5e6601902df7cddd8cc8aa0b135c77b2 (Use a bitmap in extents_t to speed up search.).
2017-03-25 13:46:56 +08:00
test_bitmap_xfu_body(&binfo, nbits); \
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
NBITS_TAB
#undef NB
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Refactor tests. Refactor tests to use explicit testing assertions, rather than diff'ing test output. This makes the test code a bit shorter, more explicitly encodes testing intent, and makes test failure diagnosis more straightforward.
2013-12-09 12:52:21 +08:00
TEST_END
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
int
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
main(void) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return test(
Implement BITMAP_INFO_INITIALIZER(nbits). This allows static initialization of bitmap_info_t structures.
2016-04-07 09:30:02 +08:00
test_bitmap_initializer,
Refactor tests. Refactor tests to use explicit testing assertions, rather than diff'ing test output. This makes the test code a bit shorter, more explicitly encodes testing intent, and makes test failure diagnosis more straightforward.
2013-12-09 12:52:21 +08:00
test_bitmap_size,
test_bitmap_init,
test_bitmap_set,
test_bitmap_unset,
Fix BITMAP_USE_TREE version of bitmap_ffu(). This fixes an extent searching regression on 32-bit systems, caused by the initial bitmap_ffu() implementation in c8021d01f6efe14dc1bd200021a815638063cb5f (Implement bitmap_ffu(), which finds the first unset bit.), as first used in 5d33233a5e6601902df7cddd8cc8aa0b135c77b2 (Use a bitmap in extents_t to speed up search.).
2017-03-25 13:46:56 +08:00
test_bitmap_xfu);
Use bitmaps to track small regions. The previous free list implementation, which embedded singly linked lists in available regions, had the unfortunate side effect of causing many cache misses during thread cache fills. Fix this in two places: - arena_run_t: Use a new bitmap implementation to track which regions are available. Furthermore, revert to preferring the lowest available region (as jemalloc did with its old bitmap-based approach). - tcache_t: Move read-only tcache_bin_t metadata into tcache_bin_info_t, and add a contiguous array of pointers to tcache_t in order to track cached objects. This substantially increases the size of tcache_t, but results in much higher data locality for common tcache operations. As a side benefit, it is again possible to efficiently flush the least recently used cached objects, so this change changes flushing from MRU to LRU. The new bitmap implementation uses a multi-level summary approach to make finding the lowest available region very fast. In practice, bitmaps only have one or two levels, though the implementation is general enough to handle extremely large bitmaps, mainly so that large page sizes can still be entertained. Fix tcache_bin_flush_large() to always flush statistics, in the same way that tcache_bin_flush_small() was recently fixed. Use JEMALLOC_DEBUG rather than NDEBUG. Add dassert(), and use it for debug-only asserts.
2011-03-17 01:30:13 +08:00
}
Reference in New Issue Copy Permalink