project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
04942c3d90
server-skynet-source-3rd-je.../test/unit/rtree.c

298 lines
7.2 KiB
C
Raw Normal View History

Add rtree unit tests.
2014-01-03 08:08:28 +08:00
#include "test/jemalloc_test.h"
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_alloc_t *rtree_node_alloc_orig;
rtree_node_dalloc_t *rtree_node_dalloc_orig;
rtree_t *test_rtree;
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
static rtree_elm_t *
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_alloc_intercept(tsdn_t *tsdn, rtree_t *rtree, size_t nelms)
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
{
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
rtree_elm_t *node;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
if (rtree != test_rtree)
return rtree_node_alloc_orig(tsdn, rtree, nelms);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
node = (rtree_elm_t *)calloc(nelms, sizeof(rtree_elm_t));
assert_ptr_not_null(node, "Unexpected calloc() failure");
return (node);
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
}
static void
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_dalloc_intercept(tsdn_t *tsdn, rtree_t *rtree, rtree_elm_t *node)
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
{
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
if (rtree != test_rtree) {
rtree_node_dalloc_orig(tsdn, rtree, node);
return;
}
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
free(node);
}
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
TEST_BEGIN(test_rtree_read_empty)
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
{
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
unsigned i;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn = tsdn_fetch();
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (i = 1; i <= (sizeof(uintptr_t) << 3); i++) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
rtree_t rtree;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx = RTREE_CTX_INITIALIZER;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
test_rtree = &rtree;
assert_false(rtree_new(&rtree, i),
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
"Unexpected rtree_new() failure");
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_ptr_null(rtree_read(tsdn, &rtree, &rtree_ctx, 0, false),
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
"rtree_read() should return NULL for empty tree");
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete(tsdn, &rtree);
test_rtree = NULL;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
}
TEST_END
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
#define NTHREADS 8
#define MAX_NBITS 18
#define NITERS 1000
#define SEED 42
typedef struct {
unsigned nbits;
rtree_t rtree;
uint32_t seed;
} thd_start_arg_t;
static void *
thd_start(void *varg)
{
thd_start_arg_t *arg = (thd_start_arg_t *)varg;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx = RTREE_CTX_INITIALIZER;
sfmt_t *sfmt;
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
extent_t *extent;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
unsigned i;
sfmt = init_gen_rand(arg->seed);
extent = (extent_t *)malloc(sizeof(extent));
assert_ptr_not_null(extent, "Unexpected malloc() failure");
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn = tsdn_fetch();
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
for (i = 0; i < NITERS; i++) {
uintptr_t key = (uintptr_t)gen_rand64(sfmt);
if (i % 2 == 0) {
rtree_elm_t *elm;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
elm = rtree_elm_acquire(tsdn, &arg->rtree, &rtree_ctx,
key, false, true);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
assert_ptr_not_null(elm,
"Unexpected rtree_elm_acquire() failure");
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_elm_write_acquired(tsdn, &arg->rtree, elm,
extent);
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
rtree_elm_release(tsdn, &arg->rtree, elm);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
elm = rtree_elm_acquire(tsdn, &arg->rtree, &rtree_ctx,
key, true, false);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
assert_ptr_not_null(elm,
"Unexpected rtree_elm_acquire() failure");
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
rtree_elm_read_acquired(tsdn, &arg->rtree, elm);
rtree_elm_release(tsdn, &arg->rtree, elm);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
} else
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_read(tsdn, &arg->rtree, &rtree_ctx, key, false);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
}
free(extent);
fini_gen_rand(sfmt);
return (NULL);
}
TEST_BEGIN(test_rtree_concurrent)
{
thd_start_arg_t arg;
thd_t thds[NTHREADS];
sfmt_t *sfmt;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
unsigned i, j;
sfmt = init_gen_rand(SEED);
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn = tsdn_fetch();
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
for (i = 1; i < MAX_NBITS; i++) {
arg.nbits = i;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
test_rtree = &arg.rtree;
assert_false(rtree_new(&arg.rtree, arg.nbits),
"Unexpected rtree_new() failure");
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
arg.seed = gen_rand32(sfmt);
for (j = 0; j < NTHREADS; j++)
thd_create(&thds[j], thd_start, (void *)&arg);
for (j = 0; j < NTHREADS; j++)
thd_join(thds[j], NULL);
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete(tsdn, &arg.rtree);
test_rtree = NULL;
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
}
fini_gen_rand(sfmt);
}
TEST_END
#undef NTHREADS
#undef MAX_NBITS
#undef NITERS
#undef SEED
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
TEST_BEGIN(test_rtree_extrema)
{
unsigned i;
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_t extent_a, extent_b;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
tsdn = tsdn_fetch();
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (i = 1; i <= (sizeof(uintptr_t) << 3); i++) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
rtree_t rtree;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx = RTREE_CTX_INITIALIZER;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
test_rtree = &rtree;
assert_false(rtree_new(&rtree, i),
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
"Unexpected rtree_new() failure");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_false(rtree_write(tsdn, &rtree, &rtree_ctx, 0,
&extent_a), "Unexpected rtree_write() failure, i=%u", i);
assert_ptr_eq(rtree_read(tsdn, &rtree, &rtree_ctx, 0, true),
&extent_a,
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
"rtree_read() should return previously set value, i=%u", i);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_false(rtree_write(tsdn, &rtree, &rtree_ctx,
~((uintptr_t)0), &extent_b),
"Unexpected rtree_write() failure, i=%u", i);
assert_ptr_eq(rtree_read(tsdn, &rtree, &rtree_ctx,
~((uintptr_t)0), true), &extent_b,
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
"rtree_read() should return previously set value, i=%u", i);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete(tsdn, &rtree);
test_rtree = NULL;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
}
TEST_END
TEST_BEGIN(test_rtree_bits)
{
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
unsigned i, j, k;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn = tsdn_fetch();
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (i = 1; i < (sizeof(uintptr_t) << 3); i++) {
uintptr_t keys[] = {0, 1,
(((uintptr_t)1) << (sizeof(uintptr_t)*8-i)) - 1};
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_t extent;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
rtree_t rtree;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx = RTREE_CTX_INITIALIZER;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
test_rtree = &rtree;
assert_false(rtree_new(&rtree, i),
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
"Unexpected rtree_new() failure");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (j = 0; j < sizeof(keys)/sizeof(uintptr_t); j++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_false(rtree_write(tsdn, &rtree, &rtree_ctx,
keys[j], &extent),
"Unexpected rtree_write() failure");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (k = 0; k < sizeof(keys)/sizeof(uintptr_t); k++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_ptr_eq(rtree_read(tsdn, &rtree,
&rtree_ctx, keys[k], true), &extent,
"rtree_read() should return previously set "
"value and ignore insignificant key bits; "
"i=%u, j=%u, k=%u, set key=%#"FMTxPTR", "
"get key=%#"FMTxPTR, i, j, k, keys[j],
keys[k]);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_ptr_null(rtree_read(tsdn, &rtree, &rtree_ctx,
Avoid atomic operations for dependent rtree reads.
2015-05-16 08:02:30 +08:00
(((uintptr_t)1) << (sizeof(uintptr_t)*8-i)), false),
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
"Only leftmost rtree leaf should be set; "
"i=%u, j=%u", i, j);
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_clear(tsdn, &rtree, &rtree_ctx, keys[j]);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete(tsdn, &rtree);
test_rtree = NULL;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
}
TEST_END
TEST_BEGIN(test_rtree_random)
{
unsigned i;
sfmt_t *sfmt;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
#define NSET 16
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
#define SEED 42
sfmt = init_gen_rand(SEED);
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn = tsdn_fetch();
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (i = 1; i <= (sizeof(uintptr_t) << 3); i++) {
uintptr_t keys[NSET];
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_t extent;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
unsigned j;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
rtree_t rtree;
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_ctx_t rtree_ctx = RTREE_CTX_INITIALIZER;
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
rtree_elm_t *elm;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
test_rtree = &rtree;
assert_false(rtree_new(&rtree, i),
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
"Unexpected rtree_new() failure");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (j = 0; j < NSET; j++) {
keys[j] = (uintptr_t)gen_rand64(sfmt);
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
elm = rtree_elm_acquire(tsdn, &rtree, &rtree_ctx,
keys[j], false, true);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
assert_ptr_not_null(elm,
"Unexpected rtree_elm_acquire() failure");
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
rtree_elm_write_acquired(tsdn, &rtree, elm, &extent);
rtree_elm_release(tsdn, &rtree, elm);
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_ptr_eq(rtree_read(tsdn, &rtree, &rtree_ctx,
keys[j], true), &extent,
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
"rtree_read() should return previously set value");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
for (j = 0; j < NSET; j++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_ptr_eq(rtree_read(tsdn, &rtree, &rtree_ctx,
keys[j], true), &extent,
"rtree_read() should return previously set value, "
"j=%u", j);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
for (j = 0; j < NSET; j++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree_clear(tsdn, &rtree, &rtree_ctx, keys[j]);
assert_ptr_null(rtree_read(tsdn, &rtree, &rtree_ctx,
keys[j], true),
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
"rtree_read() should return previously set value");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
for (j = 0; j < NSET; j++) {
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
assert_ptr_null(rtree_read(tsdn, &rtree, &rtree_ctx,
keys[j], true),
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
"rtree_read() should return previously set value");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete(tsdn, &rtree);
test_rtree = NULL;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
fini_gen_rand(sfmt);
#undef NSET
#undef SEED
}
TEST_END
int
main(void)
{
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_alloc_orig = rtree_node_alloc;
rtree_node_alloc = rtree_node_alloc_intercept;
rtree_node_dalloc_orig = rtree_node_dalloc;
rtree_node_dalloc = rtree_node_dalloc_intercept;
test_rtree = NULL;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
return (test(
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
test_rtree_read_empty,
test_rtree_concurrent,
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
test_rtree_extrema,
test_rtree_bits,
test_rtree_random));
}
Reference in New Issue Copy Permalink