Commit Graph

66 Commits

Author SHA1 Message Date
Jason Evans
a09f55c87d Wrap strerror_r().
Create the buferror() function, which wraps strerror_r().  This is
necessary because glibc provides a non-standard strerror_r().
2010-09-20 16:05:41 -07:00
Jason Evans
28177d466f Remove bad assertions in malloc_{pre,post}fork().
Remove assertions that malloc_{pre,post}fork() are only called if
threading is enabled.  This was true of these functions in the context
of FreeBSD's libc, but now the functions are called unconditionally as a
result of registering them with pthread_atfork().
2010-09-20 11:24:24 -07:00
Jason Evans
8e3c3c61b5 Add {,r,s,d}allocm().
Add allocm(), rallocm(), sallocm(), and dallocm(), which are a
functional superset of malloc(), calloc(), posix_memalign(),
malloc_usable_size(), and free().
2010-09-17 15:46:18 -07:00
Jason Evans
7e11b389aa Move size class table to man page.
Move the table of size classes from jemalloc.c to the manual page.  When
manually formatting the manual page, it is now necessary to use:

    nroff -man -t jemalloc.3
2010-09-11 22:52:16 -07:00
Jason Evans
2dbecf1f62 Port to Mac OS X.
Add Mac OS X support, based in large part on the OS X support in
Mozilla's version of jemalloc.
2010-09-11 18:20:16 -07:00
Jason Evans
2541e1b083 Add a missing mutex unlock in malloc_init_hard().
If multiple threads race to initialize malloc, the loser(s) busy-wait
until initialization is complete.  Add a missing mutex lock so that the
loser(s) properly release the initialization mutex.  Under some
race conditions, this flaw could have caused one or more threads to
become permanently blocked.

Reported by Terrell Magee.
2010-07-22 11:35:59 -07:00
Jason Evans
ed3d152ea0 Fix next_arena initialization.
If there is more than one arena, initialize next_arena so that the
first and second threads to allocate memory use arenas 0 and 1, rather
than both using arena 0.
2010-05-11 12:00:22 -07:00
Jason Evans
5065156f3f Fix threads-related profiling bugs.
Initialize bt2cnt_tsd so that cleanup at thread exit actually happens.

Associate (prof_ctx_t *) with allocated objects, rather than
(prof_thr_cnt_t *).  Each thread must always operate on its own
(prof_thr_cnt_t *), and an object may outlive the thread that allocated it.
2010-04-13 21:17:11 -07:00
Jason Evans
f18c982001 Add sampling activation/deactivation control.
Add the E/e options to control whether the application starts with
sampling active/inactive (secondary control to F/f).  Add the
prof.active mallctl so that the application can activate/deactivate
sampling on the fly.
2010-03-31 18:43:24 -07:00
Jason Evans
a02fc08ec9 Make interval-triggered profile dumping optional.
Make it possible to disable interval-triggered profile dumping, even if
profiling is enabled.  This is useful if the user only wants a single
dump at exit, or if the application manually triggers profile dumps.
2010-03-31 17:35:51 -07:00
Jason Evans
0b270a991d Reduce statistical heap sampling memory overhead.
If the mean heap sampling interval is larger than one page, simulate
sampled small objects with large objects.  This allows profiling context
pointers to be omitted for small objects.  As a result, the memory
overhead for sampling decreases as the sampling interval is increased.

Fix a compilation error in the profiling code.
2010-03-31 16:45:04 -07:00
Jason Evans
dafde14e08 Remove medium size classes.
Remove medium size classes, because concurrent dirty page purging is
no longer capable of purging inactive dirty pages inside active runs
(due to recent arena/bin locking changes).

Enhance tcache to support caching large objects, so that the same range
of size classes is still cached, despite the removal of medium size
class support.
2010-03-17 16:27:39 -07:00
Jason Evans
3fa9a2fad8 Simplify tcache object caching.
Use chains of cached objects, rather than using arrays of pointers.

Since tcache_bin_t is no longer dynamically sized, convert tcache_t's
tbin to an array of structures, rather than an array of pointers.  This
implicitly removes tcache_bin_{create,destroy}(), which further
simplifies the fast path for malloc/free.

Use cacheline alignment for tcache_t allocations.

Remove runtime configuration option for number of tcache bin slots, and
replace it with a boolean option for enabling/disabling tcache.

Limit the number of tcache objects to the lesser of TCACHE_NSLOTS_MAX
and 2X the number of regions per run for the size class.

For GC-triggered flush, discard 3/4 of the objects below the low water
mark, rather than 1/2.
2010-03-13 20:38:18 -08:00
Jason Evans
698805c525 Simplify malloc_message().
Rather than passing four strings to malloc_message(), malloc_write4(),
and all the functions that use them, only pass one string.
2010-03-03 17:45:38 -08:00
Jason Evans
b9477e782b Implement sampling for heap profiling. 2010-03-01 20:15:26 -08:00
Jason Evans
376b1529a3 Restructure source tree. 2010-02-11 14:45:59 -08:00
Jason Evans
fe5faa2cc5 Remove tracing (--enable-trace).
Remove all functionality related to tracing.  This functionality was
useful for understanding memory fragmentation during early algorithmic
design of jemalloc, but it had little utility for non-trivial
applications, due to the sheer volume of data written to disk.
2010-02-11 13:38:12 -08:00
Jason Evans
d34f9e7e93 Implement interval-based heap profile dumping.
Add mallctl interfaces for profiling parameters.

Fix a file descriptor leak in heap profile dumping.
2010-02-11 13:19:21 -08:00
Jason Evans
3383af6c2d Fix a profiling bootstrap bug.
Bootstrap profiling in three stages, so that it is usable by the time
the first application allocation occurs.
2010-02-11 08:59:06 -08:00
Jason Evans
6109fe07a1 Implement allocation profiling and leack checking.
Add the --enable-prof and --enable-prof-libunwind configure options.

Add the B/b, F/f, I/i, L/l, and U/u JEMALLOC_OPTIONS.

Interval-based profile dump triggering is not yet implemented.

Add supporting generic code:
* Add memory barriers.
* Add prn (LCG PRNG).
* Add hash (Murmur hash function).
* Add ckh (cuckoo hash tables).
2010-02-10 10:37:57 -08:00
Jason Evans
a0bf242230 Fix bootstrapping crash.
If a custom small_size2bin table was required due to non-default size
class settings, memory allocation prior to initializing chunk parameters
would cause a crash due to division by 0.  The fix re-orders the various
*_boot() function calls.

Bootstrapping is simpler now than it was before the base allocator
started just using the chunk allocator directly.  This allows
arena_boot[01]() to be combined.

Add error detection for pthread_atfork() and atexit() function calls.
2010-01-29 14:30:41 -08:00
Jason Evans
d8f565f239 Remove tcache bin sorting during flush.
This feature caused significant performance degradation, and the
fragmentation reduction benefits were difficult to quantify.
2010-01-29 13:37:31 -08:00
Jason Evans
95833311f1 madvise(..., MADV_{RANODOM,NOSYNC}) swap files.
Initialize malloc before calling into the ctl_*() functions.
2010-01-27 13:47:28 -08:00
Jason Evans
3c2343518c Implement mallctl{nametomib,bymib}().
Replace chunk stats code that was missing locking; this fixes a race
condition that could corrupt chunk statistics.

Converting malloc_stats_print() to use mallctl*().

Add a missing semicolon in th DSS code.

Convert malloc_tcache_flush() to a mallctl.

Convert malloc_swap_enable() to a set of mallctl's.
2010-01-27 13:10:56 -08:00
Jason Evans
fbbb624fc1 Simplify malloc_{pre,post}fork().
Revert to simpler lock acquistion/release code in
malloc_{pre,post}fork(), since dynamic arena rebalancing is no longer
implemented.
2010-01-24 17:56:48 -08:00
Jason Evans
4201af0542 Add the --enable-swap configure option.
Add malloc_swap_enable().

Add the O/o JEMALLOC_OPTIONS flags, which control memory overcommit.

Fix mapped memory stats reporting for arenas.
2010-01-24 02:53:40 -08:00
Jason Evans
ed1bf457fb Add the w4opaque argument to numerous functions.
Add the w4opaque argument malloc_message() and malloc_stats_print(), and
propagate the change through all the internal APIs as necessary.
2010-01-19 12:11:25 -08:00
Jason Evans
00b5c93347 Add the write4 parameter to malloc_stats_print().
Add malloc_cprintf() and malloc_vcprintf().
2010-01-17 15:49:25 -08:00
Jason Evans
b0fd5016db Add the --with-install-suffix configure option.
Fix some bugs in the Makefile's install target.
2010-01-17 01:49:20 -08:00
Jason Evans
e476f8a161 Refactor jemalloc.c into multiple source files.
Fix a stats bug in large object curruns accounting.

Replace tcache_bin_fill() with arena_tcache_fill(), and fix a bug in an OOM
error path.

Fix API name mangling to coexist with __attribute__((malloc)).
2010-01-16 09:53:50 -08:00
Jason Evans
64bd7661a8 Support malloc() even after tcache destruction has occurred, since other tsd
destructors may run after tcache_tsd's.
2010-01-12 18:13:39 -08:00
Jason Evans
e9db6c92ab Add the S/s JEMALLOC_OPTIONS flags to enable/disable tcache bin sorting. 2010-01-03 16:17:52 -08:00
Jason Evans
279e09d1ff Enhance the H/h MALLOC_OPTIONS flags to control the number of tcache bin slots,
rather than just enabling/disabling the tcache.

Fix an off-by-one bug in large object stats recording.
2010-01-03 16:16:10 -08:00
Jason Evans
3f3ecfb8e8 Enhance the G/g MALLOC_OPTIONS flags to control GC sweep interval, rather than
just enabling/disabling GC.
2010-01-03 14:45:26 -08:00
Jason Evans
952b7d192b Split run deallocation out of arena_dalloc_bin(), into arena_dalloc_bin_run().
Enhance bin run deallocation to avoid marking all pages as dirty, since the
dirty bits are already correct for all but the first page, due to the use of
arena_run_rc_{incr,decr}().  This tends to dramatically reduce the number of
pages that are marked dirty.

Modify arena_bin_run_size_calc() to assure that bin run headers never exceed
one page.  In practice, this can't happen unless hard-coded constants (related
to RUN_MAX_OVRHD) are modified, but the dirty page tracking code assumes bin
run headers never extend past the first page, so it seems worth making this a
universally valid assumption.
2010-01-03 12:10:47 -08:00
Jason Evans
03c2237af3 Make malloc_stats_print() a public symbol, and make it repeatedly callable.
Incrementally merge tcache stats, so that malloc_stats_print() always reports
at least most events that have occurred thus far.
2010-01-03 12:10:42 -08:00
Jason Evans
9ad48230ed Simplify JEMALLOC_ATTR_* macros to become JEMALLOC_ATTR().
Use JEMALLOC_ATTR(tls_model("initial-exec)) instead of -ftls-model=initial-exec
so that libjemalloc_pic.a can be directly linked into another library without
needing linker options changes.

Add attributes to malloc, calloc, and posix_memalign, for compatibility with
glibc's declarations.

Add function prototypes for the standard malloc(3) API.
2010-01-03 11:59:20 -08:00
Jason Evans
0b2368a6a2 Add the COPYING file, and consolidate copyright/licensing information. 2010-01-03 11:59:14 -08:00
Jason Evans
e888879e9c Implement per size class stats collection for large objects. 2009-12-29 00:09:15 -08:00
Jason Evans
45c128d1a9 Base maximum dirty page count on proportion of active memory.
Add dirty page tracking for pages within active small/medium object runs.

Reduce chunks_dirty red-black tree operations via lazy updating.
2009-12-29 00:09:15 -08:00
Jason Evans
94ad2b57c3 Use "lg" prefix rather than "2pow" suffix to indicate base 2 logarithms.
Remove the default definition for SIZEOF_INT_2POW.
2009-12-29 00:09:15 -08:00
Jason Evans
d64d4448cd Modify pthread_create() wrapper such that isthreaded is only set once. 2009-12-29 00:09:15 -08:00
Jason Evans
f251814779 Handle 0-size allocation requests in posix_memalign(). 2009-12-29 00:09:15 -08:00
Jason Evans
84cbbcb90a Convert thread-specific caching from magazines, and implement incremental GC.
Add the 'G'/'g' and 'H'/'h' MALLOC_OPTIONS flags.

Add the malloc_tcache_flush() function.

Disable thread-specific caching until the application goes multi-threaded.
2009-12-29 00:09:15 -08:00
Jason Evans
b2378168a4 Add support for medium size classes, [4KiB..32KiB], 2KiB apart by default.
Add the 'M' and 'm' MALLOC_OPTIONS flags, which control the maximum medium size
class.

Relax the cap on small/medium run size to arena_maxclass.

Reduce arena_run_reg_dalloc() integer division code complexity.

Increase the default chunk size from 1MiB to 4MiB.
2009-12-29 00:09:15 -08:00
Jason Evans
6d7bb5357a Remove opt_dss and opt_mmap, such that if DSS allocation is enabled, both DSS
and heap allocation are always enabled.
2009-12-29 00:09:15 -08:00
Jason Evans
5463a5298d Streamline the fast path for magazine-based allocation/deallocation. 2009-12-29 00:09:15 -08:00
Jason Evans
3ee7a5c5b0 Remove the dynamic rebalancing code, since magazines reduce its utility. 2009-12-29 00:09:15 -08:00
Jason Evans
569432cffd Implement thread-aware allocation event tracing. 2009-12-29 00:09:15 -08:00
Jason Evans
90895cf859 Add --with-jemalloc-prefix, which supports API mangling.
Rename jemalloc_options-->malloc_options and jemalloc_message-->malloc_message.
2009-12-29 00:09:15 -08:00