project-base/server-skynet-source-3rd-jemalloc
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
master
server-skynet-source-3rd-je.../test/unit/background_thread_enable.c

97 lines
3.0 KiB
C
Raw Permalink Normal View History

Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
#include "test/jemalloc_test.h"
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
const char *malloc_conf = "background_thread:false,narenas:1,max_background_threads:20";
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
Make test/unit/background_thread_enable more conservative. To avoid resource exhaustion on 32-bit platforms.
2022-05-05 02:38:57 +08:00
static unsigned
max_test_narenas(void) {
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
/*
* 10 here is somewhat arbitrary, except insofar as we want to ensure
* that the number of background threads is smaller than the number of
* arenas. I'll ragequit long before we have to spin up 10 threads per
* cpu to handle background purging, so this is a conservative
* approximation.
*/
Make test/unit/background_thread_enable more conservative. To avoid resource exhaustion on 32-bit platforms.
2022-05-05 02:38:57 +08:00
unsigned ret = 10 * ncpus;
/* Limit the max to avoid VM exhaustion on 32-bit . */
if (ret > 512) {
ret = 512;
}
return ret;
}
TEST_BEGIN(test_deferred) {
test_skip_if(!have_background_thread);
unsigned id;
size_t sz_u = sizeof(unsigned);
for (unsigned i = 0; i < max_test_narenas(); i++) {
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("arenas.create", &id, &sz_u, NULL, 0), 0,
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
"Failed to create arena");
}
bool enable = true;
size_t sz_b = sizeof(bool);
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("background_thread", NULL, NULL, &enable, sz_b), 0,
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
"Failed to enable background threads");
Fix a background_thread shutdown issue. 1) make sure background thread 0 is always created; and 2) fix synchronization between thread 0 and the control thread.
2018-03-31 06:09:05 +08:00
enable = false;
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("background_thread", NULL, NULL, &enable, sz_b), 0,
Fix a background_thread shutdown issue. 1) make sure background thread 0 is always created; and 2) fix synchronization between thread 0 and the control thread.
2018-03-31 06:09:05 +08:00
"Failed to disable background threads");
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
}
TEST_END
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
TEST_BEGIN(test_max_background_threads) {
test_skip_if(!have_background_thread);
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
size_t max_n_thds;
size_t opt_max_n_thds;
size_t sz_m = sizeof(max_n_thds);
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("opt.max_background_threads",
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
&opt_max_n_thds, &sz_m, NULL, 0), 0,
"Failed to get opt.max_background_threads");
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("max_background_threads", &max_n_thds, &sz_m, NULL,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
0), 0, "Failed to get max background threads");
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_zu_eq(opt_max_n_thds, max_n_thds,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
"max_background_threads and "
"opt.max_background_threads should match");
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("max_background_threads", NULL, NULL, &max_n_thds,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
sz_m), 0, "Failed to set max background threads");
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
unsigned id;
size_t sz_u = sizeof(unsigned);
Make test/unit/background_thread_enable more conservative. To avoid resource exhaustion on 32-bit platforms.
2022-05-05 02:38:57 +08:00
for (unsigned i = 0; i < max_test_narenas(); i++) {
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("arenas.create", &id, &sz_u, NULL, 0), 0,
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
"Failed to create arena");
}
bool enable = true;
size_t sz_b = sizeof(bool);
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("background_thread", NULL, NULL, &enable, sz_b), 0,
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
"Failed to enable background threads");
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_zu_eq(n_background_threads, max_n_thds,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
"Number of background threads should not change.\n");
size_t new_max_thds = max_n_thds - 1;
if (new_max_thds > 0) {
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("max_background_threads", NULL, NULL,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
&new_max_thds, sz_m), 0,
"Failed to set max background threads");
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_zu_eq(n_background_threads, new_max_thds,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
"Number of background threads should decrease by 1.\n");
}
new_max_thds = 1;
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_d_eq(mallctl("max_background_threads", NULL, NULL, &new_max_thds,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
sz_m), 0, "Failed to set max background threads");
Change assert_* to expect_* in tests ``` grep -Irl assert_ test/ | xargs sed -i \ 's/witness_assert/witness_do_not_replace/g'; grep -Irl assert_ test/ | xargs sed -i \ 's/malloc_mutex_assert_owner/malloc_mutex_do_not_replace_owner/g'; grep -Ir assert_ test/ | grep -o "[_a-zA-Z]*assert_[_a-zA-Z]*" | \ grep -v "^assert_"; # confirm no output grep -Irl assert_ test/ | xargs sed -i 's/assert_/expect_/g'; grep -Irl witness_do_not_replace test/ | xargs sed -i \ 's/witness_do_not_replace/witness_assert/g'; grep -Irl malloc_mutex_do_not_replace_owner test/ | xargs sed -i \ 's/malloc_mutex_do_not_replace_owner/malloc_mutex_assert_owner/g'; ```
2020-02-19 06:39:06 +08:00
expect_zu_eq(n_background_threads, new_max_thds,
Fix the max_background_thread test. We may set number of background threads separately, e.g. through --with-malloc-conf, so avoid assuming the default number in the test.
2018-05-10 07:17:37 +08:00
"Number of background threads should be 1.\n");
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
}
TEST_END
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
int
main(void) {
return test_no_reentrancy(
background_thread: add max thread count config Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
2018-03-30 03:58:13 +08:00
test_deferred,
test_max_background_threads);
Background threads: fix an indexing bug. We have a buffer overrun that manifests in the case where arena indices higher than the number of CPUs are accessed before arena indices lower than the number of CPUs. This fixes the bug and adds a test.
2018-02-27 09:11:29 +08:00
}
Reference in New Issue Copy Permalink