Commit Graph

181 Commits

Author SHA1 Message Date
Jason Evans
8f0e0eb1c0 Fix a memory corruption bug in chunk_alloc_dss().
Fix a memory corruption bug in chunk_alloc_dss() that was due to
claiming newly allocated memory is zeroed.

Reverse order of preference between mmap() and sbrk() to prefer mmap().

Clean up management of 'zero' parameter in chunk_alloc*().
2012-04-21 13:33:48 -07:00
Jason Evans
bedceea2a8 Fix isthreaded-related build breakage. 2012-04-20 14:12:30 -07:00
Jason Evans
918d6e20b7 Add missing private namespace mangling. 2012-04-20 13:42:21 -07:00
Jason Evans
7d20fbc44a Don't mangle pthread_create().
Don't mangle pthread_create(); it's an exported symbol when defined.
2012-04-20 13:06:39 -07:00
Jason Evans
f7088e6c99 Make arena_salloc() an inline function. 2012-04-19 18:28:03 -07:00
Mike Hommey
13067ec835 Remove extra argument for malloc_tsd_cleanup_register
Bookkeeping an extra argument that actually only stores a function pointer
for a function we already have is not very useful.
2012-04-18 19:25:01 -07:00
Mike Hommey
8ad483fe60 Remove initialization of the non-TLS tsd wrapper from static memory
Using static memory when malloc_tsd_malloc fails means all threads share
the same wrapper and thus the same wrapped value. This defeats the purpose
of TSD.
2012-04-18 19:23:53 -07:00
Mike Hommey
7ff1ce4131 Initialize all members of non-TLS tsd wrapper when creating it
Not setting the initialized member leads to randomly calling the cleanup
function in cases it shouldn't be called (and isn't called in other
implementations).
2012-04-18 19:23:32 -07:00
Jason Evans
86e58583bb Make special FreeBSD function overrides visible.
Make special FreeBSD libc/libthr function overrides for
_malloc_prefork(), _malloc_postfork(), and _malloc_thread_cleanup()
visible.
2012-04-18 19:01:00 -07:00
Mike Hommey
666c5bf7a8 Add a pages_purge function to wrap madvise(JEMALLOC_MADV_PURGE) calls
This will be used to implement the feature on mingw, which doesn't have
madvise.
2012-04-18 18:57:48 -07:00
Jason Evans
0b25fe79aa Update prof defaults to match common usage.
Change the "opt.lg_prof_sample" default from 0 to 19 (1 B to 512 KiB).

Change the "opt.prof_accum" default from true to false.

Add the "opt.prof_final" mallctl, so that "opt.prof_prefix" need not be
abused to disable final profile dumping.
2012-04-17 16:39:33 -07:00
Jason Evans
b57d3ec571 Add atomic(9) implementations of atomic operations.
Add atomic(9) implementations of atomic operations.  These are used on
FreeBSD for non-x86 architectures.
2012-04-17 13:27:39 -07:00
Mike Hommey
45f208e112 Replace fprintf with malloc_printf in tests. 2012-04-16 23:05:39 -07:00
Mike Hommey
72ca7220f2 Use echo instead of cat in loops in size_classes.sh
This avoids fork/exec()ing in loops, as echo is a builtin, and makes
size_classes.sh much faster (from > 10s to < 0.2s on mingw on my machine).
2012-04-16 22:45:09 -07:00
Jason Evans
1dbfd5a209 Add/remove missing/cruft entries to/from private_namespace.h. 2012-04-14 00:16:02 -07:00
Jason Evans
7ca0fdfb85 Disable munmap() if it causes VM map holes.
Add a configure test to determine whether common mmap()/munmap()
patterns cause VM map holes, and only use munmap() to discard unused
chunks if the problem does not exist.

Unify the chunk caching for mmap and dss.

Fix options processing to limit lg_chunk to be large enough that
redzones will always fit.
2012-04-12 20:20:58 -07:00
Mike Hommey
83c324acd8 Use a stub replacement and disable dss when sbrk is not supported 2012-04-12 08:43:08 -07:00
Jason Evans
5ff709c264 Normalize aligned allocation algorithms.
Normalize arena_palloc(), chunk_alloc_mmap_slow(), and
chunk_recycle_dss() to use the same algorithm for trimming
over-allocation.

Add the ALIGNMENT_ADDR2BASE(), ALIGNMENT_ADDR2OFFSET(), and
ALIGNMENT_CEILING() macros, and use them where appropriate.

Remove the run_size_p parameter from sa2u().

Fix a potential deadlock in chunk_recycle_dss() that was introduced by
eae269036c (Add alignment support to
chunk_alloc()).
2012-04-11 18:13:45 -07:00
Jason Evans
122449b073 Implement Valgrind support, redzones, and quarantine.
Implement Valgrind support, as well as the redzone and quarantine
features, which help Valgrind detect memory errors.  Redzones are only
implemented for small objects because the changes necessary to support
redzones around large and huge objects are complicated by in-place
reallocation, to the point that it isn't clear that the maintenance
burden is worth the incremental improvement to Valgrind support.

Merge arena_salloc() and arena_salloc_demote().

Refactor i[v]salloc() to expose the 'demote' option.
2012-04-11 11:46:18 -07:00
Mike Hommey
eae269036c Add alignment support to chunk_alloc(). 2012-04-10 14:51:39 -07:00
Mike Hommey
c5851eaf6e Remove MAP_NORESERVE support
It was only used by the swap feature, and that is gone.
2012-04-10 12:05:27 -07:00
Jason Evans
fad100bc35 Remove arena_malloc_prechosen().
Remove arena_malloc_prechosen(), now that arena_malloc() can be invoked
in a way that is semantically equivalent.
2012-04-06 12:24:46 -07:00
Jason Evans
b147611b52 Add utrace(2)-based tracing (--enable-utrace). 2012-04-05 13:36:17 -07:00
Jason Evans
bbe53b1c16 Revert "Use ffsl() in ALLOCM_ALIGN()."
This reverts commit 722b370399.

Unfortunately, glibc requires _GNU_SOURCE to be defined before including
string.h, but there is no reliable way to get the prototype within
jemalloc.h unless _GNU_SOURCE was already defined.
2012-04-04 15:24:01 -07:00
Jason Evans
3cc1f1aa69 Add tls_model configuration.
The tls_model attribute isn't supporte by clang (yet?), so add a
configure test that defines JEMALLOC_TLS_MODEL appropriately.
2012-04-03 22:30:05 -07:00
Jason Evans
01b3fe55ff Add a0malloc(), a0calloc(), and a0free().
Add a0malloc(), a0calloc(), and a0free(), which are used by FreeBSD's
libc to allocate/deallocate TLS in static binaries.
2012-04-03 19:25:48 -07:00
Jason Evans
633aaff967 Postpone mutex initialization on FreeBSD.
Postpone mutex initialization on FreeBSD until after base allocation is
safe.
2012-04-03 19:25:30 -07:00
Jason Evans
12a6845b6c Use $((...)) instead of expr.
Use $((...)) for math in size_classes.h rather than expr, because it is
much faster.  This is not supported syntax in the classic Bourne shell,
but all modern sh implementations support it, including bash, zsh, and
ash.
2012-04-03 13:20:21 -07:00
Jason Evans
ae4c7b4b40 Clean up *PAGE* macros.
s/PAGE_SHIFT/LG_PAGE/g and s/PAGE_SIZE/PAGE/g.

Remove remnants of the dynamic-page-shift code.

Rename the "arenas.pagesize" mallctl to "arenas.page".

Remove the "arenas.chunksize" mallctl, which is redundant with
"opt.lg_chunk".
2012-04-02 07:04:34 -07:00
Jason Evans
f004737267 Revert "Avoid NULL check in free() and malloc_usable_size()."
This reverts commit 96d4120ac0.

ivsalloc() depends on chunks_rtree being initialized.  This can be
worked around via a NULL pointer check.  However,
thread_allocated_tsd_get() also depends on initialization having
occurred, and there is no way to guard its call in free() that is
cheaper than checking whether ptr is NULL.
2012-04-02 15:18:24 -07:00
Jason Evans
96d4120ac0 Avoid NULL check in free() and malloc_usable_size().
Generalize isalloc() to handle NULL pointers in such a way that the NULL
checking overhead is only paid when introspecting huge allocations (or
NULL).  This allows free() and malloc_usable_size() to no longer check
for NULL.

Submitted by Igor Bukanov and Mike Hommey.
2012-04-02 14:50:03 -07:00
Mike Hommey
80b25932ca Move last bit of zone initialization in zone.c, and lazy-initialize 2012-04-02 14:15:20 -07:00
Jason Evans
722b370399 Use ffsl() in ALLOCM_ALIGN().
Use ffsl() rather than ffs() plus bitshifting in ALLOCM_ALIGN().  The
original rational for using ffs() was portability, but the bitmap code
has since induced a hard dependency on ffsl().
2012-04-02 14:09:07 -07:00
Jason Evans
4eeb52f080 Remove vsnprintf() and strtoumax() validation.
Remove code that validates malloc_vsnprintf() and malloc_strtoumax()
against their namesakes.  The validation code has adequately served its
usefulness at this point, and it isn't worth dealing with the different
formatting for %p with glibc versus other implementations for NULL
pointers ("(nil)" vs. "0x0").

Reported by Mike Hommey.
2012-04-02 02:30:24 -07:00
Jason Evans
f2296deb57 Clean up tsd (no functional changes). 2012-03-30 12:36:52 -07:00
Jason Evans
09a0769ba7 Work around TLS deallocation via free().
glibc uses memalign()/free() to allocate/deallocate TLS, which means
that it is unsafe to set TLS variables as a side effect of free() --
they may already be deallocated.  Work around this by avoiding
tcache_create() within free().

Reported by Mike Hommey.
2012-03-30 12:11:03 -07:00
Mike Hommey
71a93b8725 Move zone registration to zone.c 2012-03-30 10:53:00 -07:00
Mike Hommey
1a0e777024 Add a SYS_write definition on systems where it is not defined in headers
Namely, in the Android NDK headers, SYS_write is not defined; but
__NR_write is.
2012-03-30 10:21:41 -07:00
Jason Evans
d4be8b7b6e Add the "thread.tcache.enabled" mallctl. 2012-03-26 19:02:49 -07:00
Mike Hommey
c1e567bda0 Use __sync_add_and_fetch and __sync_sub_and_fetch when they are available
These functions may be available as inlines or as libgcc functions. In the
former case, a __GCC_HAVE_SYNC_COMPARE_AND_SWAP_n macro is defined. But we
still want to use these functions in the latter case, when we don't have
our own implementation.
2012-03-26 11:51:13 -07:00
Jason Evans
1e6138c88c Remove malloc_mutex_trylock().
Remove malloc_mutex_trylock(); it has not been used for quite some time.
2012-03-24 19:36:27 -07:00
Jason Evans
41b6afb834 Port to FreeBSD.
Use FreeBSD-specific functions (_pthread_mutex_init_calloc_cb(),
_malloc_{pre,post}fork()) to avoid bootstrapping issues due to
allocation in libc and libthr.

Add malloc_strtoumax() and use it instead of strtoul().  Disable
validation code in malloc_vsnprintf() and malloc_strtoumax() until
jemalloc is initialized.  This is necessary because locale
initialization causes allocation for both vsnprintf() and strtoumax().

Force the lazy-lock feature on in order to avoid pthread_self(),
because it causes allocation.

Use syscall(SYS_write, ...) rather than write(...), because libthr wraps
write() and causes allocation.  Without this workaround, it would not be
possible to print error messages in malloc_conf_init() without
substantially reworking bootstrapping.

Fix choose_arena_hard() to look at how many threads are assigned to the
candidate choice, rather than checking whether the arena is
uninitialized.  This bug potentially caused more arenas to be
initialized than necessary.
2012-02-02 23:09:53 -08:00
Jason Evans
6da5418ded Remove ephemeral mutexes.
Remove ephemeral mutexes from the prof machinery, and remove
malloc_mutex_destroy().  This simplifies mutex management on systems
that call malloc()/free() inside pthread_mutex_{create,destroy}().

Add atomic_*_u() for operation on unsigned values.

Fix prof_printf() to call malloc_vsnprintf() rather than
malloc_snprintf().
2012-03-23 18:05:51 -07:00
Jason Evans
06304a9785 Restructure atomic_*_z().
Restructure atomic_*_z() so that no casting within macros is necessary.
This avoids warnings when compiling with clang.
2012-03-23 16:09:56 -07:00
Jason Evans
9225a1991a Add JEMALLOC_CC_SILENCE_INIT().
Add JEMALLOC_CC_SILENCE_INIT(), which provides succinct syntax for
initializing a variable to avoid a spurious compiler warning.
2012-03-23 15:39:07 -07:00
Jason Evans
cd9a1346e9 Implement tsd.
Implement tsd, which is a TLS/TSD abstraction that uses one or both
internally.  Modify bootstrapping such that no tsd's are utilized until
allocation is safe.

Remove malloc_[v]tprintf(), and use malloc_snprintf() instead.

Fix %p argument size handling in malloc_vsnprintf().

Fix a long-standing statistics-related bug in the "thread.arena"
mallctl that could cause crashes due to linked list corruption.
2012-03-23 15:14:55 -07:00
Jason Evans
e24c7af35d Invert NO_TLS to JEMALLOC_TLS. 2012-03-19 10:21:17 -07:00
Jason Evans
6508bc6931 Remove #include <sys/sysctl.h>.
Remove #include <sys/sysctl.h>, which is no longer needed (now using
sysconf(3) to get number of CPUs).
2012-03-15 17:07:42 -07:00
Jason Evans
4e2e3dd9cf Fix fork-related bugs.
Acquire/release arena bin locks as part of the prefork/postfork.  This
bug made deadlock in the child between fork and exec a possibility.

Split jemalloc_postfork() into jemalloc_postfork_{parent,child}() so
that the child can reinitialize mutexes rather than unlocking them.  In
practice, this bug tended not to cause problems.
2012-03-13 16:31:41 -07:00
Jason Evans
824d34e5b7 Modify malloc_vsnprintf() validation code.
Modify malloc_vsnprintf() validation code to verify that output is
identical to vsnprintf() output, even if both outputs are truncated due
to buffer exhaustion.
2012-03-13 13:19:04 -07:00
Jason Evans
0a0bbf63e5 Implement aligned_alloc().
Implement aligned_alloc(), which was added in the C11 standard.  The
function is weakly specified to the point that a minimally compliant
implementation would be painful to use (size must be an integral
multiple of alignment!), which in practice makes posix_memalign() a
safer choice.
2012-03-13 12:55:21 -07:00
Jason Evans
4c2faa8a7c Fix a regression in JE_COMPILABLE().
Revert JE_COMPILABLE() so that it detects link errors.  Cross-compiling
should still work as long as a valid configure cache is provided.

Clean up some comments/whitespace.
2012-03-13 11:09:23 -07:00
Jason Evans
125b93e43f Remove bashism.
Submitted by Mike Hommey.
2012-03-12 11:33:59 -07:00
Jason Evans
d81e4bdd5c Implement malloc_vsnprintf().
Implement malloc_vsnprintf() (a subset of vsnprintf(3)) as well as
several other printing functions based on it, so that formatted printing
can be relied upon without concern for inducing a dependency on floating
point runtime support.  Replace malloc_write() calls with
malloc_*printf() where doing so simplifies the code.

Add name mangling for library-private symbols in the data and BSS
sections.  Adjust CONF_HANDLE_*() macros in malloc_conf_init() to expose
all opt_* variable use to cpp so that proper mangling occurs.
2012-03-07 16:19:19 -08:00
Jason Evans
4507f34628 Remove the lg_tcache_gc_sweep option.
Remove the lg_tcache_gc_sweep option, because it is no longer
very useful.  Prior to the addition of dynamic adjustment of tcache fill
count, it was possible for fill/flush overhead to be a problem, but this
problem no longer occurs.
2012-03-05 14:34:37 -08:00
Jason Evans
b8c8be7f8a Use UINT64_C() rather than LLU for 64-bit constants. 2012-03-05 12:26:26 -08:00
Jason Evans
3492daf1ce Add SH4 and mips architecture support.
Submitted by Andreas Vinsander.
2012-03-05 12:16:57 -08:00
Jason Evans
7e77eaffff Add the --disable-experimental option. 2012-03-02 17:47:37 -08:00
Jason Evans
84f7cdb0c5 Rename prn to prng.
Rename prn to prng so that Windows doesn't choke when trying to create
a file named prn.h.
2012-03-02 15:59:45 -08:00
Jason Evans
62320b8551 Reorder macros. 2012-03-01 17:53:16 -08:00
Jason Evans
0a5489e37d Add --with-mangling.
Add the --with-mangling configure option, which can be used to specify
name mangling on a per public symbol basis that takes precedence over
--with-jemalloc-prefix.

Expose the memalign() and valloc() overrides even if
--with-jemalloc-prefix is specified.  This change does no real harm, and
simplifies the code.
2012-03-01 17:19:20 -08:00
Jason Evans
7e15dab94d Add nallocm().
Add nallocm(), which computes the real allocation size that would result
from the corresponding allocm() call.  nallocm() is a functional
superset of OS X's malloc_good_size(), in that it takes alignment
constraints into account.
2012-02-29 12:56:37 -08:00
Jason Evans
4bb0983013 Use glibc allocator hooks.
When jemalloc is used as a libc malloc replacement (i.e. not prefixed),
some particular setups may end up inconsistently calling malloc from
libc and free from jemalloc, or the other way around.

glibc provides hooks to make its functions use alternative
implementations.  Use them.

Submitted by Karl Tomlinson and Mike Hommey.
2012-02-29 10:37:27 -08:00
Jason Evans
3add8d8cda Remove unused variables in tcache_dalloc_large().
Submitted by Mike Hommey.
2012-02-28 21:08:19 -08:00
Jason Evans
c90ad71237 Remove the sysv option. 2012-02-28 20:31:37 -08:00
Jason Evans
b172610317 Simplify small size class infrastructure.
Program-generate small size class tables for all valid combinations of
LG_TINY_MIN, LG_QUANTUM, and PAGE_SHIFT.  Use the appropriate table to generate
all relevant data structures, and remove the distinction between
tiny/quantum/cacheline/subpage bins.

Remove --enable-dynamic-page-shift.  This option didn't prove useful in
practice, and it prevented optimizations.

Add Tilera architecture support.
2012-02-28 16:50:47 -08:00
Jason Evans
5389146191 Remove the opt.lg_prof_bt_max option.
Remove opt.lg_prof_bt_max, and hard code it to 7.  The original
intention of this option was to enable faster backtracing by limiting
backtrace depth.  However, this makes graphical pprof output very
difficult to interpret.  In practice, decreasing sampling frequency is a
better mechanism for limiting profiling overhead.
2012-02-13 18:41:36 -08:00
Jason Evans
0b526ff94d Remove the opt.lg_prof_tcmax option.
Remove the opt.lg_prof_tcmax option and hard-code a cache size of 1024.
This setting is something that users just shouldn't have to worry about.
If lock contention actually ends up being a problem, the simple solution
available to the user is to reduce sampling frequency.
2012-02-13 18:04:26 -08:00
Jason Evans
746868929a Remove highruns statistics. 2012-02-13 15:18:19 -08:00
Jason Evans
ef8897b4b9 Make 8-byte tiny size class non-optional.
When tiny size class support was first added, it was intended to support
truly tiny size classes (even 2 bytes).  However, this wasn't very
useful in practice, so the minimum tiny size class has been limited to
sizeof(void *) for a long time now.  This is too small to be standards
compliant, but other commonly used malloc implementations do not even
bother using a 16-byte quantum  on systems with vector units (SSE2+,
AltiVEC, etc.).  As such, it is safe in practice to support an 8-byte
tiny size class on 64-bit systems that support 16-byte types.
2012-02-13 15:03:59 -08:00
Jason Evans
962463d9b5 Streamline tcache-related malloc/free fast paths.
tcache_get() is inlined, so do the config_tcache check inside
tcache_get() and simplify its callers.

Make arena_malloc() an inline function, since it is part of the malloc()
fast path.

Remove conditional logic that cause build issues if --disable-tcache was
specified.
2012-02-13 12:29:49 -08:00
Jason Evans
4162627757 Remove the swap feature.
Remove the swap feature, which enabled per application swap files.  In
practice this feature has not proven itself useful to users.
2012-02-13 10:56:17 -08:00
Jason Evans
fd56043c53 Remove magic.
Remove structure magic, because 1) it is no longer conditional, and 2)
it stopped being very effective at detecting memory corruption several
years ago.
2012-02-13 10:24:43 -08:00
Jason Evans
7372b15a31 Reduce cpp conditional logic complexity.
Convert configuration-related cpp conditional logic to use static
constant variables, e.g.:

  #ifdef JEMALLOC_DEBUG
    [...]
  #endif

becomes:

  if (config_debug) {
    [...]
  }

The advantage is clearer, more concise code.  The main disadvantage is
that data structures no longer have conditionally defined fields, so
they pay the cost of all fields regardless of whether they are used.  In
practice, this is only a minor concern; config_stats will go away in an
upcoming change, and config_prof is the only other major feature that
depends on more than a few special-purpose fields.
2012-02-10 20:22:09 -08:00
Jason Evans
12a4887826 Fix huge_ralloc to maintain chunk statistics.
Fix huge_ralloc() to properly maintain chunk statistics when using
mremap(2).
2011-11-11 14:41:59 -08:00
Jason Evans
da9dde0854 Clean up rb documentation. 2011-11-01 20:48:31 -07:00
Jason Evans
a507004d29 Fix off-by-one backtracing issues.
Rewrite prof_alloc_prep() as a cpp macro, PROF_ALLOC_PREP(), in order to
remove any doubt as to whether an additional stack frame is created.
Prior to this change, it was assumed that inlining would reduce the
total number of frames in the backtrace, but in practice behavior wasn't
completely predictable.

Create imemalign() and call it from posix_memalign(), memalign(), and
valloc(), so that all entry points require the same number of stack
frames to be ignored during backtracing.
2011-08-12 13:48:27 -07:00
Jason Evans
04ca1efe35 Adjust relative #include for private_namespace.h. 2011-07-30 17:58:07 -07:00
Jason Evans
746e77a06b Add the --with-private-namespace option.
Add the --with-private-namespace option to make it possible to work
around library-private symbols being exposed in static libraries.
2011-07-30 16:40:52 -07:00
Jason Evans
f0b22cf932 Use LLU suffix for all 64-bit constants.
Add the LLU suffix for all 0x... 64-bit constants.

Reported by Jakob Blomer.
2011-05-22 10:49:44 -07:00
Jason Evans
7427525c28 Move repo contents in jemalloc/ to top level. 2011-03-31 20:36:17 -07:00