project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22bc75ee3e
server-skynet-source-3rd-je.../test/unit/rtree.c

229 lines
6.8 KiB
C
Raw Normal View History

Add rtree unit tests.
2014-01-03 08:08:28 +08:00
#include "test/jemalloc_test.h"
Header refactoring: unify and de-catchall rtree module.
2017-05-24 05:26:31 +08:00
#include "jemalloc/internal/rtree.h"
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_alloc_t *rtree_node_alloc_orig;
rtree_node_dalloc_t *rtree_node_dalloc_orig;
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_alloc_t *rtree_leaf_alloc_orig;
rtree_leaf_dalloc_t *rtree_leaf_dalloc_orig;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
/* Potentially too large to safely place on the stack. */
rtree_t test_rtree;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
static rtree_node_elm_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_node_alloc_intercept(tsdn_t *tsdn, rtree_t *rtree, size_t nelms) {
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_node_elm_t *node;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
if (rtree != &test_rtree) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
return rtree_node_alloc_orig(tsdn, rtree, nelms);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
malloc_mutex_unlock(tsdn, &rtree->init_lock);
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
node = (rtree_node_elm_t *)calloc(nelms, sizeof(rtree_node_elm_t));
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
assert_ptr_not_null(node, "Unexpected calloc() failure");
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
malloc_mutex_lock(tsdn, &rtree->init_lock);
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return node;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
}
static void
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_node_dalloc_intercept(tsdn_t *tsdn, rtree_t *rtree,
rtree_node_elm_t *node) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
if (rtree != &test_rtree) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_dalloc_orig(tsdn, rtree, node);
return;
}
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
free(node);
}
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
static rtree_leaf_elm_t *
rtree_leaf_alloc_intercept(tsdn_t *tsdn, rtree_t *rtree, size_t nelms) {
rtree_leaf_elm_t *leaf;
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
if (rtree != &test_rtree) {
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
return rtree_leaf_alloc_orig(tsdn, rtree, nelms);
}
malloc_mutex_unlock(tsdn, &rtree->init_lock);
leaf = (rtree_leaf_elm_t *)calloc(nelms, sizeof(rtree_leaf_elm_t));
assert_ptr_not_null(leaf, "Unexpected calloc() failure");
malloc_mutex_lock(tsdn, &rtree->init_lock);
return leaf;
}
static void
rtree_leaf_dalloc_intercept(tsdn_t *tsdn, rtree_t *rtree,
rtree_leaf_elm_t *leaf) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
if (rtree != &test_rtree) {
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_dalloc_orig(tsdn, rtree, leaf);
return;
}
free(leaf);
}
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_rtree_read_empty) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn_t *tsdn;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
tsdn = tsdn_fetch();
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_t *rtree = &test_rtree;
Add init function support to tsd members. This will facilitate embedding tcache into tsd, which will require proper initialization cannot be done via the static initializer. Make tsd->rtree_ctx to be initialized via rtree_ctx_data_init().
2017-03-30 04:18:02 +08:00
rtree_ctx_t rtree_ctx;
rtree_ctx_data_init(&rtree_ctx);
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_new(rtree, false), "Unexpected rtree_new() failure");
assert_ptr_null(rtree_extent_read(tsdn, rtree, &rtree_ctx, PAGE,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
false), "rtree_extent_read() should return NULL for empty tree");
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_delete(tsdn, rtree);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
TEST_END
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
#undef NTHREADS
#undef NITERS
#undef SEED
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_rtree_extrema) {
Rename extent_node_t to extent_t.
2016-03-24 12:09:28 +08:00
extent_t extent_a, extent_b;
SC: Make some key size classes static. The largest small class, smallest large class, and largest large class may all be needed down fast paths; to avoid the risk of touching another cache line, we can make them available as constants.
2018-07-12 07:05:58 +08:00
extent_init(&extent_a, NULL, NULL, SC_LARGE_MINCLASS, false,
sz_size2index(SC_LARGE_MINCLASS), 0,
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
extent_state_active, false, false, true, EXTENT_NOT_HEAD);
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_init(&extent_b, NULL, NULL, 0, false, SC_NSIZES, 0,
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
extent_state_active, false, false, true, EXTENT_NOT_HEAD);
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
tsdn_t *tsdn = tsdn_fetch();
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_t *rtree = &test_rtree;
Add init function support to tsd members. This will facilitate embedding tcache into tsd, which will require proper initialization cannot be done via the static initializer. Make tsd->rtree_ctx to be initialized via rtree_ctx_data_init().
2017-03-30 04:18:02 +08:00
rtree_ctx_t rtree_ctx;
rtree_ctx_data_init(&rtree_ctx);
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_new(rtree, false), "Unexpected rtree_new() failure");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_write(tsdn, rtree, &rtree_ctx, PAGE, &extent_a,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_szind_get(&extent_a), extent_slab_get(&extent_a)),
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
"Unexpected rtree_write() failure");
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_szind_slab_update(tsdn, rtree, &rtree_ctx, PAGE,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_szind_get(&extent_a), extent_slab_get(&extent_a));
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_eq(rtree_extent_read(tsdn, rtree, &rtree_ctx, PAGE, true),
Remove rtree support for 0 (NULL) keys. NULL can never actually be inserted in practice, and removing support allows a branch to be removed from the fast path.
2017-02-05 18:50:59 +08:00
&extent_a,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
"rtree_extent_read() should return previously set value");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_write(tsdn, rtree, &rtree_ctx, ~((uintptr_t)0),
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
&extent_b, extent_szind_get_maybe_invalid(&extent_b),
extent_slab_get(&extent_b)), "Unexpected rtree_write() failure");
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_eq(rtree_extent_read(tsdn, rtree, &rtree_ctx,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
~((uintptr_t)0), true), &extent_b,
"rtree_extent_read() should return previously set value");
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_delete(tsdn, rtree);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
TEST_END
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_rtree_bits) {
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
tsdn_t *tsdn = tsdn_fetch();
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Remove rtree support for 0 (NULL) keys. NULL can never actually be inserted in practice, and removing support allows a branch to be removed from the fast path.
2017-02-05 18:50:59 +08:00
uintptr_t keys[] = {PAGE, PAGE + 1,
PAGE + (((uintptr_t)1) << LG_PAGE) - 1};
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
extent_t extent;
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_init(&extent, NULL, NULL, 0, false, SC_NSIZES, 0,
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
extent_state_active, false, false, true, EXTENT_NOT_HEAD);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_t *rtree = &test_rtree;
Add init function support to tsd members. This will facilitate embedding tcache into tsd, which will require proper initialization cannot be done via the static initializer. Make tsd->rtree_ctx to be initialized via rtree_ctx_data_init().
2017-03-30 04:18:02 +08:00
rtree_ctx_t rtree_ctx;
rtree_ctx_data_init(&rtree_ctx);
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_new(rtree, false), "Unexpected rtree_new() failure");
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
for (unsigned i = 0; i < sizeof(keys)/sizeof(uintptr_t); i++) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_write(tsdn, rtree, &rtree_ctx, keys[i],
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
&extent, SC_NSIZES, false),
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
"Unexpected rtree_write() failure");
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
for (unsigned j = 0; j < sizeof(keys)/sizeof(uintptr_t); j++) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_eq(rtree_extent_read(tsdn, rtree, &rtree_ctx,
keys[j], true), &extent,
"rtree_extent_read() should return previously set "
"value and ignore insignificant key bits; i=%u, "
"j=%u, set key=%#"FMTxPTR", get key=%#"FMTxPTR, i,
j, keys[i], keys[j]);
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
}
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_null(rtree_extent_read(tsdn, rtree, &rtree_ctx,
Remove rtree support for 0 (NULL) keys. NULL can never actually be inserted in practice, and removing support allows a branch to be removed from the fast path.
2017-02-05 18:50:59 +08:00
(((uintptr_t)2) << LG_PAGE), false),
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
"Only leftmost rtree leaf should be set; i=%u", i);
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_clear(tsdn, rtree, &rtree_ctx, keys[i]);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_delete(tsdn, rtree);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
TEST_END
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
TEST_BEGIN(test_rtree_random) {
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define NSET 16
#define SEED 42
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
sfmt_t *sfmt = init_gen_rand(SEED);
tsdn_t *tsdn = tsdn_fetch();
uintptr_t keys[NSET];
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_t *rtree = &test_rtree;
Add init function support to tsd members. This will facilitate embedding tcache into tsd, which will require proper initialization cannot be done via the static initializer. Make tsd->rtree_ctx to be initialized via rtree_ctx_data_init().
2017-03-30 04:18:02 +08:00
rtree_ctx_t rtree_ctx;
rtree_ctx_data_init(&rtree_ctx);
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
extent_t extent;
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
extent_init(&extent, NULL, NULL, 0, false, SC_NSIZES, 0,
Implement retain on Windows. The VirtualAlloc and VirtualFree APIs are different because MEM_DECOMMIT cannot be used across multiple VirtualAlloc regions. To properly support decommit, only allow merge / split within the same region -- this is done by tracking the "is_head" state of extents and not merging cross-region. Add a new state is_head (only relevant for retain && !maps_coalesce), which is true for the first extent in each VirtualAlloc region. Determine if two extents can be merged based on the head state, and use serial numbers for sanity checks.
2019-07-13 07:20:23 +08:00
extent_state_active, false, false, true, EXTENT_NOT_HEAD);
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_false(rtree_new(rtree, false), "Unexpected rtree_new() failure");
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
for (unsigned i = 0; i < NSET; i++) {
keys[i] = (uintptr_t)gen_rand64(sfmt);
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
rtree_leaf_elm_t *elm = rtree_leaf_elm_lookup(tsdn, rtree,
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
&rtree_ctx, keys[i], false, true);
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
assert_ptr_not_null(elm,
Protect the rtree/extent interactions with a mutex pool. Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
2017-05-16 05:23:51 +08:00
"Unexpected rtree_leaf_elm_lookup() failure");
Hide size class computation behind a layer of indirection. This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
2017-12-15 04:46:39 +08:00
rtree_leaf_elm_write(tsdn, rtree, elm, &extent, SC_NSIZES,
false);
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_eq(rtree_extent_read(tsdn, rtree, &rtree_ctx,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
keys[i], true), &extent,
"rtree_extent_read() should return previously set value");
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
}
for (unsigned i = 0; i < NSET; i++) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_eq(rtree_extent_read(tsdn, rtree, &rtree_ctx,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
keys[i], true), &extent,
"rtree_extent_read() should return previously set value, "
"i=%u", i);
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
}
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
for (unsigned i = 0; i < NSET; i++) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_clear(tsdn, rtree, &rtree_ctx, keys[i]);
assert_ptr_null(rtree_extent_read(tsdn, rtree, &rtree_ctx,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
keys[i], true),
"rtree_extent_read() should return previously set value");
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
for (unsigned i = 0; i < NSET; i++) {
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
assert_ptr_null(rtree_extent_read(tsdn, rtree, &rtree_ctx,
Incorporate szind/slab into rtree leaves. Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
2017-03-17 08:57:52 +08:00
keys[i], true),
"rtree_extent_read() should return previously set value");
Determine rtree levels at compile time. Rather than dynamically building a table to aid per level computations, define a constant table at compile time. Omit both high and low insignificant bits. Use one to three tree levels, depending on the number of significant bits.
2017-02-07 05:17:12 +08:00
}
Embed root node into rtree_t. This avoids one atomic operation per tree access.
2017-03-23 07:38:03 +08:00
rtree_delete(tsdn, rtree);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
fini_gen_rand(sfmt);
#undef NSET
#undef SEED
}
TEST_END
int
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
main(void) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_alloc_orig = rtree_node_alloc;
rtree_node_alloc = rtree_node_alloc_intercept;
rtree_node_dalloc_orig = rtree_node_dalloc;
rtree_node_dalloc = rtree_node_dalloc_intercept;
Split rtree_elm_t into rtree_{node,leaf}_elm_t. This allows leaf elements to differ in size from internal node elements. In principle it would be more correct to use a different type for each level of the tree, but due to implementation details related to atomic operations, we use casts anyway, thus counteracting the value of additional type correctness. Furthermore, such a scheme would require function code generation (via cpp macros), as well as either unwieldy type names for leaves or type aliases, e.g. typedef struct rtree_elm_d2_s rtree_leaf_elm_t; This alternate strategy would be more correct, and with less code duplication, but probably not worth the complexity.
2017-03-17 00:46:42 +08:00
rtree_leaf_alloc_orig = rtree_leaf_alloc;
rtree_leaf_alloc = rtree_leaf_alloc_intercept;
rtree_leaf_dalloc_orig = rtree_leaf_dalloc;
rtree_leaf_dalloc = rtree_leaf_dalloc_intercept;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return test(
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
test_rtree_read_empty,
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
test_rtree_extrema,
test_rtree_bits,
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
test_rtree_random);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Reference in New Issue Copy Permalink