project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
cdc240d501
server-skynet-source-3rd-je.../src/rtree.c

440 lines
11 KiB
C
Raw Normal View History

Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define JEMALLOC_RTREE_C_
Port to Mac OS X. Add Mac OS X support, based in large part on the OS X support in Mozilla's version of jemalloc.
2010-09-06 01:35:13 +08:00
#include "jemalloc/internal/jemalloc_internal.h"
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
static unsigned
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
hmin(unsigned ha, unsigned hb) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
return (ha < hb ? ha : hb);
}
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
/*
* Only the most significant bits of keys passed to rtree_{read,write}() are
* used.
*/
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
bool
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_new(rtree_t *rtree, unsigned bits) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
unsigned bits_in_leaf, height, i;
Port to Mac OS X. Add Mac OS X support, based in large part on the OS X support in Mozilla's version of jemalloc.
2010-09-06 01:35:13 +08:00
Optimize rtree_get(). Specialize fast path to avoid code that cannot execute for dependent loads. Manually unroll.
2016-03-23 08:54:35 +08:00
assert(RTREE_HEIGHT_MAX == ((ZU(1) << (LG_SIZEOF_PTR+3)) /
RTREE_BITS_PER_LEVEL));
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
assert(bits > 0 && bits <= (sizeof(uintptr_t) << 3));
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
bits_in_leaf = (bits % RTREE_BITS_PER_LEVEL) == 0 ? RTREE_BITS_PER_LEVEL
: (bits % RTREE_BITS_PER_LEVEL);
Convert rtree from (void *) to (uint8_t) storage. Reduce rtree memory usage by storing booleans (1 byte each) rather than pointers. The rtree code is only used to record whether jemalloc manages a chunk of memory, so there's no need to store pointers in the rtree. Increase rtree node size to 64 KiB in order to reduce tree depth from 13 to 3 on 64-bit systems. The conversion to more compact leaf nodes was enough by itself to make the rtree depth 1 on 32-bit systems; due to the fact that root nodes are smaller than the specified node size if possible, the node size change has no impact on 32-bit systems (assuming default chunk size).
2014-01-03 09:36:38 +08:00
if (bits > bits_in_leaf) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
height = 1 + (bits - bits_in_leaf) / RTREE_BITS_PER_LEVEL;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if ((height-1) * RTREE_BITS_PER_LEVEL + bits_in_leaf != bits) {
Convert rtree from (void *) to (uint8_t) storage. Reduce rtree memory usage by storing booleans (1 byte each) rather than pointers. The rtree code is only used to record whether jemalloc manages a chunk of memory, so there's no need to store pointers in the rtree. Increase rtree node size to 64 KiB in order to reduce tree depth from 13 to 3 on 64-bit systems. The conversion to more compact leaf nodes was enough by itself to make the rtree depth 1 on 32-bit systems; due to the fact that root nodes are smaller than the specified node size if possible, the node size change has no impact on 32-bit systems (assuming default chunk size).
2014-01-03 09:36:38 +08:00
height++;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
} else {
Convert rtree from (void *) to (uint8_t) storage. Reduce rtree memory usage by storing booleans (1 byte each) rather than pointers. The rtree code is only used to record whether jemalloc manages a chunk of memory, so there's no need to store pointers in the rtree. Increase rtree node size to 64 KiB in order to reduce tree depth from 13 to 3 on 64-bit systems. The conversion to more compact leaf nodes was enough by itself to make the rtree depth 1 on 32-bit systems; due to the fact that root nodes are smaller than the specified node size if possible, the node size change has no impact on 32-bit systems (assuming default chunk size).
2014-01-03 09:36:38 +08:00
height = 1;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
assert((height-1) * RTREE_BITS_PER_LEVEL + bits_in_leaf == bits);
rtree->height = height;
/* Root level. */
rtree->levels[0].subtree = NULL;
rtree->levels[0].bits = (height > 1) ? RTREE_BITS_PER_LEVEL :
bits_in_leaf;
rtree->levels[0].cumbits = rtree->levels[0].bits;
/* Interior levels. */
for (i = 1; i < height-1; i++) {
rtree->levels[i].subtree = NULL;
rtree->levels[i].bits = RTREE_BITS_PER_LEVEL;
rtree->levels[i].cumbits = rtree->levels[i-1].cumbits +
RTREE_BITS_PER_LEVEL;
Convert rtree from (void *) to (uint8_t) storage. Reduce rtree memory usage by storing booleans (1 byte each) rather than pointers. The rtree code is only used to record whether jemalloc manages a chunk of memory, so there's no need to store pointers in the rtree. Increase rtree node size to 64 KiB in order to reduce tree depth from 13 to 3 on 64-bit systems. The conversion to more compact leaf nodes was enough by itself to make the rtree depth 1 on 32-bit systems; due to the fact that root nodes are smaller than the specified node size if possible, the node size change has no impact on 32-bit systems (assuming default chunk size).
2014-01-03 09:36:38 +08:00
}
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
/* Leaf level. */
Convert rtree from (void *) to (uint8_t) storage. Reduce rtree memory usage by storing booleans (1 byte each) rather than pointers. The rtree code is only used to record whether jemalloc manages a chunk of memory, so there's no need to store pointers in the rtree. Increase rtree node size to 64 KiB in order to reduce tree depth from 13 to 3 on 64-bit systems. The conversion to more compact leaf nodes was enough by itself to make the rtree depth 1 on 32-bit systems; due to the fact that root nodes are smaller than the specified node size if possible, the node size change has no impact on 32-bit systems (assuming default chunk size).
2014-01-03 09:36:38 +08:00
if (height > 1) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
rtree->levels[height-1].subtree = NULL;
rtree->levels[height-1].bits = bits_in_leaf;
rtree->levels[height-1].cumbits = bits;
}
Port to Mac OS X. Add Mac OS X support, based in large part on the OS X support in Mozilla's version of jemalloc.
2010-09-06 01:35:13 +08:00
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
/* Compute lookup table to be used by rtree_[ctx_]start_level(). */
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
for (i = 0; i < RTREE_HEIGHT_MAX; i++) {
rtree->start_level[i] = hmin(RTREE_HEIGHT_MAX - 1 - i, height -
1);
Port to Mac OS X. Add Mac OS X support, based in large part on the OS X support in Mozilla's version of jemalloc.
2010-09-06 01:35:13 +08:00
}
Add rtree lookup path caching. rtree-based extent lookups remain more expensive than chunk-based run lookups, but with this optimization the fast path slowdown is ~3 CPU cycles per metadata lookup (on Intel Core i7-4980HQ), versus ~11 cycles prior. The path caching speedup tends to degrade gracefully unless allocated memory is spread far apart (as is the case when using a mixture of sbrk() and mmap()).
2016-06-03 09:43:10 +08:00
rtree->start_level[RTREE_HEIGHT_MAX] = 0;
Port to Mac OS X. Add Mac OS X support, based in large part on the OS X support in Mozilla's version of jemalloc.
2010-09-06 01:35:13 +08:00
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
malloc_mutex_init(&rtree->init_lock, "rtree", WITNESS_RANK_RTREE);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return false;
Port to Mac OS X. Add Mac OS X support, based in large part on the OS X support in Mozilla's version of jemalloc.
2010-09-06 01:35:13 +08:00
}
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
#ifdef JEMALLOC_JET
#undef rtree_node_alloc
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define rtree_node_alloc JEMALLOC_N(rtree_node_alloc_impl)
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
#endif
static rtree_elm_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_node_alloc(tsdn_t *tsdn, rtree_t *rtree, size_t nelms) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return (rtree_elm_t *)base_alloc(tsdn, b0get(), nelms *
sizeof(rtree_elm_t), CACHELINE);
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
}
#ifdef JEMALLOC_JET
#undef rtree_node_alloc
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define rtree_node_alloc JEMALLOC_N(rtree_node_alloc)
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_alloc_t *rtree_node_alloc = JEMALLOC_N(rtree_node_alloc_impl);
#endif
#ifdef JEMALLOC_JET
#undef rtree_node_dalloc
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define rtree_node_dalloc JEMALLOC_N(rtree_node_dalloc_impl)
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
#endif
UNUSED static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_node_dalloc(tsdn_t *tsdn, rtree_t *rtree, rtree_elm_t *node) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
/* Nodes are never deleted during normal operation. */
not_reached();
}
#ifdef JEMALLOC_JET
#undef rtree_node_dalloc
Replace tabs following #define with spaces. This resolves #564.
2017-01-20 13:41:41 +08:00
#define rtree_node_dalloc JEMALLOC_N(rtree_node_dalloc)
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_dalloc_t *rtree_node_dalloc = JEMALLOC_N(rtree_node_dalloc_impl);
#endif
#ifdef JEMALLOC_JET
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
static void
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete_subtree(tsdn_t *tsdn, rtree_t *rtree, rtree_elm_t *node,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
unsigned level) {
Fix unsigned comparison underflow. These bugs only affected tests and debug builds.
2015-03-12 14:14:50 +08:00
if (level + 1 < rtree->height) {
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
size_t nchildren, i;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
nchildren = ZU(1) << rtree->levels[level].bits;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
for (i = 0; i < nchildren; i++) {
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
rtree_elm_t *child = node[i].child;
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
if (child != NULL) {
rtree_delete_subtree(tsdn, rtree, child, level +
1);
}
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
}
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_dalloc(tsdn, rtree, node);
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_delete(tsdn_t *tsdn, rtree_t *rtree) {
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
unsigned i;
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
for (i = 0; i < rtree->height; i++) {
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
rtree_elm_t *subtree = rtree->levels[i].subtree;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (subtree != NULL) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_delete_subtree(tsdn, rtree, subtree, i);
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
}
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
}
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
#endif
Add rtree unit tests.
2014-01-03 08:08:28 +08:00
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
static rtree_elm_t *
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
rtree_node_init(tsdn_t *tsdn, rtree_t *rtree, unsigned level,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_elm_t **elmp) {
Add element acquire/release capabilities to rtree. This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
2016-03-28 18:06:35 +08:00
rtree_elm_t *node;
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
malloc_mutex_lock(tsdn, &rtree->init_lock);
node = atomic_read_p((void**)elmp);
if (node == NULL) {
Refactor rtree to always use base_alloc() for node allocation.
2016-04-16 15:36:11 +08:00
node = rtree_node_alloc(tsdn, rtree, ZU(1) <<
rtree->levels[level].bits);
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
if (node == NULL) {
malloc_mutex_unlock(tsdn, &rtree->init_lock);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return NULL;
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
}
Refactor rtree to be lock-free. Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
2015-01-31 14:54:08 +08:00
atomic_write_p((void **)elmp, node);
}
Fix long spinning in rtree_node_init rtree_node_init spinlocks the node, allocates, and then sets the node. This is under heavy contention at the top of the tree if many threads start to allocate at the same time. Instead, take a per-rtree sleeping mutex to reduce spinning. Tested both pthreads and osx OSSpinLock, and both reduce spinning adequately Previous benchmark time: ./ttest1 500 100 ~15s New benchmark time: ./ttest1 500 100 .57s
2016-11-01 07:23:33 +08:00
malloc_mutex_unlock(tsdn, &rtree->init_lock);
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return node;
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
}
Make non-essential inline rtree functions static functions.
2017-02-04 11:44:33 +08:00
static unsigned
rtree_start_level(const rtree_t *rtree, uintptr_t key) {
unsigned start_level;
if (unlikely(key == 0)) {
return rtree->height - 1;
}
start_level = rtree->start_level[(lg_floor(key) + 1) >>
LG_RTREE_BITS_PER_LEVEL];
assert(start_level < rtree->height);
return start_level;
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
}
Make non-essential inline rtree functions static functions.
2017-02-04 11:44:33 +08:00
static bool
rtree_node_valid(rtree_elm_t *node) {
return ((uintptr_t)node != (uintptr_t)0);
}
static rtree_elm_t *
rtree_child_tryread(rtree_elm_t *elm, bool dependent) {
rtree_elm_t *child;
/* Double-checked read (first read may be stale). */
child = elm->child;
if (!dependent && !rtree_node_valid(child)) {
child = (rtree_elm_t *)atomic_read_p(&elm->pun);
}
assert(!dependent || child != NULL);
return child;
}
static rtree_elm_t *
rtree_child_read(tsdn_t *tsdn, rtree_t *rtree, rtree_elm_t *elm, unsigned level,
bool dependent) {
rtree_elm_t *child;
child = rtree_child_tryread(elm, dependent);
if (!dependent && unlikely(!rtree_node_valid(child))) {
child = rtree_node_init(tsdn, rtree, level+1, &elm->child);
}
assert(!dependent || child != NULL);
return child;
}
static rtree_elm_t *
rtree_subtree_tryread(rtree_t *rtree, unsigned level, bool dependent) {
rtree_elm_t *subtree;
/* Double-checked read (first read may be stale). */
subtree = rtree->levels[level].subtree;
if (!dependent && unlikely(!rtree_node_valid(subtree))) {
subtree = (rtree_elm_t *)atomic_read_p(
&rtree->levels[level].subtree_pun);
}
assert(!dependent || subtree != NULL);
return subtree;
}
static rtree_elm_t *
rtree_subtree_read(tsdn_t *tsdn, rtree_t *rtree, unsigned level,
bool dependent) {
rtree_elm_t *subtree;
subtree = rtree_subtree_tryread(rtree, level, dependent);
if (!dependent && unlikely(!rtree_node_valid(subtree))) {
subtree = rtree_node_init(tsdn, rtree, level,
&rtree->levels[level].subtree);
}
assert(!dependent || subtree != NULL);
return subtree;
Fix fork(2)-related deadlocks. Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
2012-10-10 05:46:22 +08:00
}
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
Split rtree_elm_lookup_hard() out of rtree_elm_lookup(). Anything but a hit in the first element of the lookup cache is expensive enough to negate the benefits of inlining.
2017-02-04 12:12:49 +08:00
rtree_elm_t *
rtree_elm_lookup_hard(tsdn_t *tsdn, rtree_t *rtree, rtree_ctx_t *rtree_ctx,
uintptr_t key, bool dependent, bool init_missing) {
unsigned start_level = rtree_start_level(rtree, key);
rtree_elm_t *node = init_missing ? rtree_subtree_read(tsdn, rtree,
start_level, dependent) : rtree_subtree_tryread(rtree, start_level,
dependent);
#define RTREE_GET_BIAS (RTREE_HEIGHT_MAX - rtree->height)
switch (start_level + RTREE_GET_BIAS) {
#define RTREE_GET_SUBTREE(level) \
case level: { \
assert(level < (RTREE_HEIGHT_MAX-1)); \
if (!dependent && unlikely(!rtree_node_valid(node))) { \
return NULL; \
} \
uintptr_t subkey = rtree_subkey(rtree, key, level - \
RTREE_GET_BIAS); \
node = init_missing ? rtree_child_read(tsdn, rtree, \
&node[subkey], level - RTREE_GET_BIAS, dependent) : \
rtree_child_tryread(&node[subkey], dependent); \
/* Fall through. */ \
}
#define RTREE_GET_LEAF(level) \
case level: { \
assert(level == (RTREE_HEIGHT_MAX-1)); \
if (!dependent && unlikely(!rtree_node_valid(node))) { \
return NULL; \
} \
/* \
* node is a leaf, so it contains values rather than \
* child pointers. \
*/ \
if (likely(key != 0)) { \
if (RTREE_CTX_NCACHE > 1) { \
memmove(&rtree_ctx->cache[1], \
&rtree_ctx->cache[0], \
sizeof(rtree_ctx_cache_elm_t) * \
(RTREE_CTX_NCACHE-1)); \
} \
uintptr_t leafkey = rtree_leafkey(rtree, key); \
rtree_ctx->cache[0].leafkey = leafkey; \
rtree_ctx->cache[0].leaf = node; \
} \
uintptr_t subkey = rtree_subkey(rtree, key, level - \
RTREE_GET_BIAS); \
return &node[subkey]; \
}
#if RTREE_HEIGHT_MAX > 1
RTREE_GET_SUBTREE(0)
#endif
#if RTREE_HEIGHT_MAX > 2
RTREE_GET_SUBTREE(1)
#endif
#if RTREE_HEIGHT_MAX > 3
RTREE_GET_SUBTREE(2)
#endif
#if RTREE_HEIGHT_MAX > 4
RTREE_GET_SUBTREE(3)
#endif
#if RTREE_HEIGHT_MAX > 5
RTREE_GET_SUBTREE(4)
#endif
#if RTREE_HEIGHT_MAX > 6
RTREE_GET_SUBTREE(5)
#endif
#if RTREE_HEIGHT_MAX > 7
RTREE_GET_SUBTREE(6)
#endif
#if RTREE_HEIGHT_MAX > 8
RTREE_GET_SUBTREE(7)
#endif
#if RTREE_HEIGHT_MAX > 9
RTREE_GET_SUBTREE(8)
#endif
#if RTREE_HEIGHT_MAX > 10
RTREE_GET_SUBTREE(9)
#endif
#if RTREE_HEIGHT_MAX > 11
RTREE_GET_SUBTREE(10)
#endif
#if RTREE_HEIGHT_MAX > 12
RTREE_GET_SUBTREE(11)
#endif
#if RTREE_HEIGHT_MAX > 13
RTREE_GET_SUBTREE(12)
#endif
#if RTREE_HEIGHT_MAX > 14
RTREE_GET_SUBTREE(13)
#endif
#if RTREE_HEIGHT_MAX > 15
RTREE_GET_SUBTREE(14)
#endif
#if RTREE_HEIGHT_MAX > 16
# error Unsupported RTREE_HEIGHT_MAX
#endif
RTREE_GET_LEAF(RTREE_HEIGHT_MAX-1)
#undef RTREE_GET_SUBTREE
#undef RTREE_GET_LEAF
default: not_reached();
}
#undef RTREE_GET_BIAS
not_reached();
}
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
static int
rtree_elm_witness_comp(const witness_t *a, void *oa, const witness_t *b,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
void *ob) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
uintptr_t ka = (uintptr_t)oa;
uintptr_t kb = (uintptr_t)ob;
assert(ka != 0);
assert(kb != 0);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return (ka > kb) - (ka < kb);
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
}
static witness_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_elm_witness_alloc(tsd_t *tsd, uintptr_t key, const rtree_elm_t *elm) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness_t *witness;
size_t i;
rtree_elm_witness_tsd_t *witnesses = tsd_rtree_elm_witnessesp_get(tsd);
/* Iterate over entire array to detect double allocation attempts. */
witness = NULL;
for (i = 0; i < sizeof(rtree_elm_witness_tsd_t) / sizeof(witness_t);
i++) {
rtree_elm_witness_t *rew = &witnesses->witnesses[i];
assert(rew->elm != elm);
if (rew->elm == NULL && witness == NULL) {
rew->elm = elm;
witness = &rew->witness;
witness_init(witness, "rtree_elm",
WITNESS_RANK_RTREE_ELM, rtree_elm_witness_comp,
(void *)key);
}
}
assert(witness != NULL);
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return witness;
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
}
static witness_t *
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_elm_witness_find(tsd_t *tsd, const rtree_elm_t *elm) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
size_t i;
rtree_elm_witness_tsd_t *witnesses = tsd_rtree_elm_witnessesp_get(tsd);
for (i = 0; i < sizeof(rtree_elm_witness_tsd_t) / sizeof(witness_t);
i++) {
rtree_elm_witness_t *rew = &witnesses->witnesses[i];
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (rew->elm == elm) {
Remove extraneous parens around return arguments. This resolves #540.
2017-01-20 10:15:45 +08:00
return &rew->witness;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
}
not_reached();
}
static void
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
rtree_elm_witness_dalloc(tsd_t *tsd, witness_t *witness,
const rtree_elm_t *elm) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
size_t i;
rtree_elm_witness_tsd_t *witnesses = tsd_rtree_elm_witnessesp_get(tsd);
for (i = 0; i < sizeof(rtree_elm_witness_tsd_t) / sizeof(witness_t);
i++) {
rtree_elm_witness_t *rew = &witnesses->witnesses[i];
if (rew->elm == elm) {
rew->elm = NULL;
witness_init(&rew->witness, "rtree_elm",
WITNESS_RANK_RTREE_ELM, rtree_elm_witness_comp,
NULL);
return;
}
}
not_reached();
}
void
rtree_elm_witness_acquire(tsdn_t *tsdn, const rtree_t *rtree, uintptr_t key,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
const rtree_elm_t *elm) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness_t *witness;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (tsdn_null(tsdn)) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness = rtree_elm_witness_alloc(tsdn_tsd(tsdn), key, elm);
witness_lock(tsdn, witness);
}
void
rtree_elm_witness_access(tsdn_t *tsdn, const rtree_t *rtree,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
const rtree_elm_t *elm) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness_t *witness;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (tsdn_null(tsdn)) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness = rtree_elm_witness_find(tsdn_tsd(tsdn), elm);
witness_assert_owner(tsdn, witness);
}
void
rtree_elm_witness_release(tsdn_t *tsdn, const rtree_t *rtree,
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
const rtree_elm_t *elm) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness_t *witness;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
if (tsdn_null(tsdn)) {
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
return;
Update brace style. Add braces around single-line blocks, and remove line breaks before function-opening braces. This resolves #537.
2017-01-16 08:56:30 +08:00
}
Add rtree element witnesses.
2016-04-18 03:55:10 +08:00
witness = rtree_elm_witness_find(tsdn_tsd(tsdn), elm);
witness_unlock(tsdn, witness);
rtree_elm_witness_dalloc(tsdn_tsd(tsdn), witness, elm);
}
Reference in New Issue Copy Permalink