Commit Graph

1827 Commits

Author SHA1 Message Date
Jason Evans
596b479d83 Skip default tcache testing if !opt_tcache. 2017-06-01 08:55:27 -07:00
David Goldblatt
fa35463d56 Witness assertions: only assert locklessness when non-reentrant.
Previously we could still hit these assertions down error paths or in the
extended API.
2017-05-31 17:02:54 -07:00
Qi Wang
508f54b02b Use real pthread_create for creating background threads. 2017-05-31 16:48:13 -07:00
Jason Evans
9a86c9bd30 Clean source directory before building tests in object directories. 2017-05-31 15:07:30 -07:00
David Goldblatt
8261e581be Header refactoring: Pull size helpers out of jemalloc module. 2017-05-31 13:08:45 -07:00
David Goldblatt
041e041e1f Header refactoring: unify and de-catchall mutex_pool. 2017-05-31 13:08:45 -07:00
David Goldblatt
98774e64a4 Header refactoring: unify and de-catchall extent_mmap module. 2017-05-31 13:08:45 -07:00
David Goldblatt
93284bb53d Header refactoring: unify and de-catchall extent_dss. 2017-05-31 13:08:45 -07:00
David Goldblatt
44f9bd147a Header refactoring: unify and de-catchall rtree module. 2017-05-31 13:08:45 -07:00
David Goldblatt
b4b4a98bc8 Add /run_tests.out/ to .gitignore. 2017-05-31 10:37:48 -07:00
Jason Evans
10d090aae9 Pass the O_CLOEXEC flag to open(2).
This resolves #528.
2017-05-31 08:50:35 -07:00
Qi Wang
66813916b5 Track background thread status separately at fork.
Use a separate boolean to track the enabled status, instead of leaving the
global background thread status inconsistent.
2017-05-31 08:27:31 -07:00
Qi Wang
2e4d1a4e30 Output total_wait_ns for bin mutexes. 2017-05-30 22:25:11 -07:00
Jason Evans
ff8062a511 Add jemalloc prefix to allocator functions pruned by jeprof.
This resolves #507.
2017-05-30 20:22:00 -07:00
Qi Wang
7578b0e929 Explicitly say so when aborting on opt_abort_conf. 2017-05-30 17:37:35 -07:00
Jason Evans
685c97fc43 More thoroughly document the *.{nmalloc,ndalloc,nrequests} mallctls.
This resolves #412.
2017-05-30 15:58:36 -07:00
Jason Evans
c606a87d2a Add the --disable-thp option to support cross compiling.
This resolves #669.
2017-05-30 11:30:54 -07:00
Qi Wang
bf6673a070 Fix npages during arena_decay_epoch_advance().
We do not lock extents while advancing epoch.  This change makes sure that we
only read npages from extents once in order to avoid any inconsistency.
2017-05-30 10:26:53 -07:00
Jason Evans
4f0963b883 Add test for excessive retained memory. 2017-05-29 17:27:18 -07:00
Jason Evans
168793a1c1 Fix extent_grow_next management.
Fix management of extent_grow_next to serialize operations that may grow
retained memory.  This assures that the sizes of the newly allocated
extents correspond to the size classes in the intended growth sequence.

Fix management of extent_grow_next to skip size classes if a request is
too large to be satisfied by the next size in the growth sequence.  This
avoids the potential for an arbitrary number of requests to bypass
triggering extent_grow_next increases.

This resolves #858.
2017-05-29 17:27:18 -07:00
Jason Evans
a16114866a Fix OOM paths in extent_grow_retained(). 2017-05-29 17:27:18 -07:00
Qi Wang
d5ef5ae934 Add opt.stats_print_opts.
The value is passed to atexit(3)-triggered malloc_stats_print() calls.
2017-05-29 11:54:00 -07:00
Qi Wang
49505e558b Make test/unit/background_thread not flaky. 2017-05-26 21:15:15 -07:00
Qi Wang
b86d271cbf Added opt_abort_conf: abort on invalid config options. 2017-05-26 21:14:28 -07:00
Jason Evans
57aaa53f2b Fix run_tests to avoid percpu_arena on !Linux. 2017-05-26 16:17:01 -07:00
Qi Wang
927239b910 Cleanup smoothstep.sh / .h.
h_step_sum was used to compute moving sum.  Not in use anymore.
2017-05-25 16:52:10 -07:00
Qi Wang
1df18d7c83 Fix stats.mapped during deallocation. 2017-05-24 15:57:46 -07:00
Jason Evans
67c93c332a Refactor run_tests to increase parallelism.
Rather than relying on parallel make to build individual configurations
one at a time, use xargs to build multiple configurations in parallel.
This allows the configure scripts to run in parallel.  On a 14-core
system (28 hyperthreads), this increases average CPU utilization from
~20% to ~90%.
2017-05-24 15:55:19 -07:00
David Goldblatt
18ecbfa89e Header refactoring: unify and de-catchall mutex module 2017-05-24 15:27:30 -07:00
David Goldblatt
9f822a1fd7 Header refactoring: unify and de-catchall witness code. 2017-05-24 15:27:30 -07:00
Jason Evans
36195c8f4d Disable percpu_arena by default. 2017-05-23 15:32:50 -07:00
Jason Evans
196a53c2ae Do not assume dss never decreases.
An sbrk() caller outside jemalloc can decrease the dss, so add a
separate atomic boolean to explicitly track whether jemalloc is
concurrently calling sbrk(), rather than depending on state outside
jemalloc's full control.

This resolves #802.
2017-05-23 15:31:29 -07:00
Jason Evans
067b970130 Add dss:primary testing.
Generalize the run_tests.sh and .travis.yml test generation to handle
combinations of arguments to the --with-malloc-conf configure option,
and merge "dss:primary" into the existing "tcache:false" testing.
2017-05-23 15:31:29 -07:00
Jason Evans
9b1038d19c Do not hold the base mutex while calling extent hooks.
Drop the base mutex while allocating new base blocks, because extent
allocation can enter code that prohibits holding non-core mutexes, e.g.
the extent_[d]alloc() and extent_purge_forced_wrapper() calls in
extent_alloc_dss().

This partially resolves #802.
2017-05-23 15:31:29 -07:00
Qi Wang
eeefdf3ce8 Fix # of unpurged pages in decay algorithm.
When # of dirty pages move below npages_limit (e.g. they are reused), we should
not lower number of unpurged pages because that would cause the reused pages to
be double counted in the backlog (as a result, decay happen slower than it
should).  Instead, set number of unpurged to the greater of current npages and
npages_limit.

Added an assertion: the ceiling # of pages should be greater than npages_limit.
2017-05-23 13:48:30 -07:00
Qi Wang
0eae838b0d Check for background thread inactivity on extents_dalloc.
To avoid background threads sleeping forever with idle arenas, we eagerly check
background threads' sleep time after extents_dalloc, and signal the thread if
necessary.
2017-05-23 12:26:20 -07:00
Qi Wang
2c368284d2 Add tests for background threads. 2017-05-23 12:26:20 -07:00
Qi Wang
44559e7cf1 Add documentation for background_thread related options. 2017-05-23 12:26:20 -07:00
Qi Wang
5f5ed2198e Add profiling for the background thread mutex. 2017-05-23 12:26:20 -07:00
Qi Wang
2bee0c6251 Add background thread related stats. 2017-05-23 12:26:20 -07:00
Qi Wang
b693c7868e Implementing opt.background_thread.
Added opt.background_thread to enable background threads, which handles purging
currently.  When enabled, decay ticks will not trigger purging (which will be
left to the background threads).  We limit the max number of threads to NCPUs.
When percpu arena is enabled, set CPU affinity for the background threads as
well.

The sleep interval of background threads is dynamic and determined by computing
number of pages to purge in the future (based on backlog).
2017-05-23 12:26:20 -07:00
David Goldblatt
3f685e8824 Protect the rtree/extent interactions with a mutex pool.
Instead of embedding a lock bit in rtree leaf elements, we associate extents
with a small set of mutexes.  This gets us two things:

- We can use the system mutexes.  This (hypothetically) protects us from
  priority inversion, and lets us stop doing a backoff/sleep loop, instead
  opting for precise wakeups from the mutex.
- Cuts down on the number of mutex acquisitions we have to do (from 4 in the
  worst case to two).

We end up simplifying most of the rtree code (which no longer has to deal with
locking or concurrency at all), at the cost of additional complexity in the
extent code: since the mutex protecting the rtree leaf elements is determined by
reading the extent out of those elements, the initial read is racy, so that we
may acquire an out of date mutex.  We re-check the extent in the leaf after
acquiring the mutex to protect us from this race.
2017-05-19 14:21:27 -07:00
David Goldblatt
26c792e61a Allow mutexes to take a lock ordering enum at construction.
This lets us specify whether and how mutexes of the same rank are allowed to be
acquired.  Currently, we only allow two polices (only a single mutex at a given
rank at a time, and mutexes acquired in ascending order), but we can plausibly
allow more (e.g. the "release uncontended mutexes before blocking").
2017-05-19 14:21:27 -07:00
Jason Evans
6e62c62862 Refactor *decay_time into *decay_ms.
Support millisecond resolution for decay times.  Among other use cases
this makes it possible to specify a short initial dirty-->muzzy decay
phase, followed by a longer muzzy-->clean decay phase.

This resolves #812.
2017-05-18 11:33:45 -07:00
Qi Wang
baf3e294e0 Add stats: arena uptime. 2017-05-18 10:04:28 -07:00
Jason Evans
04fec5e084 Avoid over-rebuilding due to namespace mangling.
Take care not to touch generated namespace mangling headers unless their
contents would change.

This resolves #838.
2017-05-17 10:06:58 -07:00
Qi Wang
b8ba3c3132 Use srcroot path for private_namespace.sh. 2017-05-16 09:30:33 -07:00
Jason Evans
18a83681cf Refactor (MALLOCX_ARENA_MAX + 1) to be MALLOCX_ARENA_LIMIT.
This resolves #673.
2017-05-14 10:14:23 -07:00
Jason Evans
909f0482e4 Automatically generate private symbol name mangling macros.
Rather than using a manually maintained list of internal symbols to
drive name mangling, add a compilation phase to automatically extract
the list of internal symbols.

This resolves #677.
2017-05-11 23:06:54 -07:00
Jason Evans
a4ae9707da Remove unused private_unnamespace infrastructure. 2017-05-11 23:06:54 -07:00