Making C++ data races show up on purpose
A data race is the bug that passes every test on your machine and then corrupts memory on a customer’s. It does not show up because the broken interleaving is rare. Two threads have to touch the same memory in the same instant, and on a quiet dev box they almost never do.
Years ago I worked on making a large legacy C++ app thread-safe, and the team’s trick was to stop waiting for the rare case and force it. In debug builds, every UI action ran the same work on all cores at once, so a race that happened once in a million natural runs happened on almost every one. I have rebuilt that harness in every job since. This post is that harness pointed at the usual concurrency tools, one at a time, to watch each one fail when it is used the way the internet usually shows it.
Here is the whole thing.
// How many hardware threads to use, with a sane fallback for the platforms
// where hardware_concurrency() cannot tell and returns 0.
inline uint32_t hw_threads() {
uint32_t n = std::thread::hardware_concurrency();
return n ? n : 4;
}
// Call `body(thread_index, iteration)` on every hardware thread at once,
// `iters` times each, and wait for all of them to finish.
template <typename Body>
void hammer(Body body, uint32_t threads = hw_threads(), int32_t iters = 100000) {
std::atomic<bool> go{false}; // every thread waits on this
std::vector<std::thread> pool;
pool.reserve(threads);
for (uint32_t t = 0; t < threads; ++t) {
pool.emplace_back([&, t] {
while (!go.load(std::memory_order_acquire)) { /* spin */ }
for (int32_t i = 0; i < iters; ++i) body(t, i);
});
}
go.store(true, std::memory_order_release); // release them all together
for (auto& th : pool) th.join();
}It does not reason about races. It parks every thread on one flag, releases them together, and lets them all hit the shared state in the same instant. That is enough to turn most races from rare into routine.
Every output below is captured at build time by npm run verify, built with clang++ -std=c++23 on my 12-core, 24-thread Ryzen 9 3900X. The command under each snippet shows its exact flags. The numbers move from run to run, the conclusions do not.
A plain int is not safe
The smallest possible shared state is one integer that several threads increment.
int32_t counter = 0; // a plain int, shared by every thread
void increment() {
counter++; // load counter, add one, store it back: three steps,
} // and another thread can slip in between them24 threads, 100000 increments each, expected total 2400000
trial 1: counter = 100000 (lost 2300000, 96%)
trial 2: counter = 100000 (lost 2300000, 96%)
trial 3: counter = 100000 (lost 2300000, 96%)
...
lost increments in 10 of 10 trials
counter reached 4% of the expected total on average
Twenty-four threads each add one a hundred thousand times, so the counter should reach 2.4 million. It reaches a small fraction. counter++ is not one step but three: load, add one, store. While one thread holds the old value in a register, a dozen others load the same value and store back a total that is already stale. The lost increments are not a rounding error, they are most of the work.
Marking it volatile is not the fix
Search for this bug and the top answer is often “make it volatile”. So here is the exact same counter with that one word added.
volatile int32_t counter = 0; // "volatile makes it thread-safe" -- it does not
void increment() {
counter++; // volatile forces the load and the store to really
} // happen, but they are still two separate steps24 threads, 100000 increments each, expected total 2400000
trial 1: counter = 164212 (lost 2235788, 93%)
trial 2: counter = 153731 (lost 2246269, 94%)
trial 3: counter = 158503 (lost 2241497, 93%)
...
lost increments in 10 of 10 trials, exactly like the plain int
counter reached 7% of the expected total on average
It loses just as many increments. volatile was designed for memory that changes outside the program, like a hardware register or a signal handler’s flag. It tells the compiler not to cache the value in a register and not to drop the load or the store. It says nothing about other threads, and it does not make the read-modify-write happen as one step. The three steps are still three steps, and other threads still slip between them.
Atomics, and the wrong way to use them
std::atomic<int32_t> is the right type, but only a single operation on it is indivisible. The common mistake is to assume that because the variable is atomic, anything you write with it is safe. Spell the increment as a load and then a store and you are back to two operations with a gap between them.
std::atomic<int32_t> counter{0};
void bump_wrong() {
counter.store(counter.load() + 1); // two atomic ops with a gap between them:
} // another thread can store into that gap24 threads, 100000 increments each, expected total 2400000
store(load() + 1) : 191188 (lost 2208812)
It loses almost as much as the plain int did. The fix is fetch_add, which does the whole read-modify-write as one step the hardware cannot split.
void bump_right() {
counter.fetch_add(1, std::memory_order_relaxed); // one indivisible step
}24 threads, 100000 increments each, expected total 2400000
fetch_add(1) : 2400000 (lost 0)
It lands on the exact total, every time. Atomicity is a property of an operation, not of a variable.
The subtler trap with atomics is memory ordering, the thing behind most broken busy-waits. The default is sequentially consistent and always correct. Loosen it to std::memory_order_relaxed and each operation stays atomic, but the writes around it can become visible to another thread out of order. That is the one race the harness cannot force, because it hides on the x86 this runs on and only wakes up on a weaker memory model like a phone’s ARM core, so it is the example for ThreadSanitizer below.
A mutex makes it correct, and adds a new way to fail
A mutex sidesteps the question of which operations are atomic. Hold the lock, touch the shared state, release it.
std::mutex m;
{
std::lock_guard<std::mutex> lk(m); // locked here
counter++; // any code is safe inside
} // unlocked here, even on an exceptionThat is correct under any number of threads. The new failure mode is the deadlock. Take two mutexes in one order on one thread and the opposite order on another, and each thread ends up holding the lock the other is waiting for. Neither ever moves.
std::mutex a, b;
void wants_a_then_b() {
std::lock_guard<std::mutex> la(a); // take a ...
std::this_thread::sleep_for(20ms); // (a window to make the deadlock certain)
std::lock_guard<std::mutex> lb(b); // ... then wait for b
}
void wants_b_then_a() {
std::lock_guard<std::mutex> lb(b); // take b ...
std::this_thread::sleep_for(20ms);
std::lock_guard<std::mutex> la(a); // ... then wait for a
}opposite lock orders (thread 1: a then b, thread 2: b then a)
DEADLOCK: nobody finished in 500 ms.
thread 1 holds a and waits for b, thread 2 holds b and waits for a
The small sleep between the two locks is only there to make the deadlock certain instead of occasional, and a watchdog turns the hang into a printed line so the program can finish. The fix is to never let two threads disagree about lock order. std::scoped_lock takes all the mutexes it is given at once, using an algorithm that cannot deadlock. The same opposing-order work, run through the harness on every thread, now never hangs.
std::mutex c, d;
void takes_both() {
std::scoped_lock lk(c, d); // locks both at once in a deadlock-free order
// ... work on whatever c and d protect ...
}two locks taken with std::scoped_lock(c, d), on every thread
finished: every thread took both locks, no deadlock in any run
Condition variables and the lost wakeup
A condition variable lets a thread sleep until something is true. The trap is to wait on the notification instead of on the condition. A notification is not stored anywhere, so one that arrives before the wait is gone for good.
std::mutex m;
std::condition_variable cv;
bool ready = false;
void consumer_wrong() {
std::this_thread::sleep_for(100ms); // arrive at the wait after the notify
std::unique_lock<std::mutex> lk(m);
cv.wait(lk); // waits for a notify -- even one that already happened
// ... use the data ...
}
void producer() {
{
std::lock_guard<std::mutex> lk(m);
ready = true;
}
cv.notify_one(); // if this runs before the wait, nobody is listening
}notify sent before the wait, cv.wait(lk) with no predicate
LOST WAKEUP: the notify happened before the wait, so the wait never returns
The producer sets the flag and notifies while the consumer is still on its way to the wait. By the time the consumer calls cv.wait(lk), the notification has already come and gone, and the wait blocks forever. The fix is the predicate overload, which checks the condition before it ever sleeps and again on every wakeup.
std::mutex m2;
std::condition_variable cv2;
bool ready2 = false;
void consumer_right() {
std::this_thread::sleep_for(100ms); // same late arrival as the broken one
std::unique_lock<std::mutex> lk(m2);
cv2.wait(lk, [] { return ready2; }); // checks the condition first, every wake
// ... use the data ...
}notify-before-wait again, cv.wait(lk, [] { return ready; })
consumer woke immediately: the predicate was already true, so it never waited
Because the flag is already true, the second consumer sees it and never waits at all. The same overload also covers spurious wakeups, where the wait returns with nothing actually ready, since it simply rechecks the predicate and goes back to sleep. The rule is to always wait on a condition, never on the signal.
ThreadSanitizer, so you do not have to get lucky
The harness makes races likely. It does not make them certain, and it can only find a race in code you thought to point it at. For the real thing there is ThreadSanitizer, built into Clang and GCC. It does not need the race to actually corrupt anything. It watches the memory accesses as they happen and reports any two that touch the same location from different threads without ordering between them.
Here is the same counter race, two threads this time, built with -fsanitize=thread.
int32_t counter = 0;
int main() {
std::thread t1([] { for (int32_t i = 0; i < 100000; ++i) counter++; });
std::thread t2([] { for (int32_t i = 0; i < 100000; ++i) counter++; });
t1.join();
t2.join();
return 0;
}WARNING: ThreadSanitizer: data race
Write of size 4 by thread T2:
tsan_counter.cpp:13
Previous write of size 4 by thread T1:
tsan_counter.cpp:12
Location is global 'counter' of size 4
SUMMARY: ThreadSanitizer: data race in tsan_counter.cpp:13
It names the conflicting write and read, the line each one is on, and the variable they fight over. The report above is trimmed to those stable lines, the full one also prints both threads’ stack traces.
It also catches the race the harness cannot force. The classic one is a broken busy-wait: spin on an atomic flag, then read the data the flag was meant to guard. Making the flag atomic feels like enough, but with relaxed ordering the flag promises nothing about the writes that came before it.
void produce_relaxed() {
data = 42; // 1. write the data
ready.store(true, std::memory_order_relaxed); // 2. announce it, relaxed
}
void consume_relaxed() {
while (!ready.load(std::memory_order_relaxed)) {} // busy-wait on the flag
int32_t got = data; // 3. read, unordered with step 1
std::print("data = {}\n", got);
}WARNING: ThreadSanitizer: data race
Read of size 4 by thread T2:
busywait.cpp:24
Previous write of size 4 by thread T1:
busywait.cpp:19
Location is global 'data' of size 4
SUMMARY: ThreadSanitizer: data race in busywait.cpp:24
ThreadSanitizer flags the read of data against the write on the other thread, even though on this x86 box the value almost always arrives in time. The fix is to publish with release and receive with acquire. That pairs the store and the load, so everything written before the store is visible after the load, and the default sequentially consistent ordering does the same. The read is now ordered after the write, and the report is gone.
void produce_release() {
data = 42;
ready.store(true, std::memory_order_release); // release: orders step 1 before it
}
void consume_acquire() {
while (!ready.load(std::memory_order_acquire)) {} // acquire: receives step 1
int32_t got = data;
std::print("data = {}\n", got);
}data = 42
(ThreadSanitizer: no data race)
This is the tool the old debug-build harness was a stand-in for. Run it in CI over a test suite that exercises the threaded paths and it catches races the harness would only find if it happened to hammer the right spot. The two go together: the harness drives many threads through the code, ThreadSanitizer watches what they touch.
A faster allocator with placement new
The second half of this post builds something real with these tools. Calling new for every small object goes through the general allocator, which can free any block in any order and stay thread-safe, and pays for all of that on every call. Plenty of code needs none of it. Allocate a heap of objects for one phase of work, throw them all away at the end, and a bump allocator is far faster: one big buffer, a cursor, and the next slice handed out on each call.
class Arena {
std::byte* base_;
std::size_t cap_, next_ = 0;
public:
explicit Arena(std::size_t bytes)
: base_(static_cast<std::byte*>(::operator new(bytes))), cap_(bytes) {}
~Arena() { ::operator delete(base_); }
void* allocate(std::size_t n) {
if (next_ + n > cap_) return nullptr; // full: time to reset()
void* p = base_ + next_;
next_ += n; // a read-modify-write, like counter++
return p;
}
void reset() { next_ = 0; } // "frees" everything at once
};There is no per-object free, and that is the point. It barely does anything per call, and giving up one-at-a-time frees is what buys the speed.
The arena hands out raw bytes. To turn those bytes into an object you use placement new, which runs a constructor at an address you already own instead of allocating. It pairs with an explicit destructor call, since there is no delete to run one for you.
struct Widget {
int32_t id;
float weight;
Widget(int32_t i, float w) : id(i), weight(w) {}
};
void example(Arena& arena) {
void* mem = arena.allocate(sizeof(Widget)); // raw bytes from the buffer
Widget* w = new (mem) Widget(7, 1.5f); // placement new: construct in place
// ... use w ...
w->~Widget(); // run the destructor by hand
// no free: the bytes come back only when the whole arena is reset()
}The cursor is a data race
next_ += n is the counter bug wearing a different hat. It is a read-modify-write on shared state, so the same harness that lost increments now hands the same slice to two threads at once.
// Each thread takes a slice, stamps its own id into it, yields, then checks the
// stamp is still its own. A surviving foreign stamp means two threads were
// handed the same slice.
template <typename A>
int64_t collisions(A& arena, uint32_t threads, int32_t iters) {
std::atomic<int64_t> hits{0};
hammer(
[&](uint32_t tid, int32_t) {
void* p = arena.allocate(64);
if (!p) return;
auto* stamp = static_cast<uint32_t*>(p);
*stamp = tid;
std::this_thread::yield();
if (*stamp != tid) hits.fetch_add(1, std::memory_order_relaxed);
},
threads, iters);
return hits.load();
}Arena (next_ += n) : 18692 slices handed to more than one thread
The fix is the one that fixed the counter. The cursor is the only mutable state, so bumping it with fetch_add is all the arena needs, and it stays lock-free. There is no compare-exchange loop and no ABA problem, because the cursor only ever moves forward, it is never rewound to memory still in use.
// The same arena, made safe to share. The only mutable state is the cursor, so
// the only fix needed is to bump it atomically. fetch_add is one instruction,
// it is lock-free, and there is no ABA problem here because the cursor only ever
// moves forward, it is never rewound to memory still in use. To stay bounded it
// wraps within a power-of-two buffer, which suits transient, touch-and-discard
// allocation (one slot is only reused a whole buffer's worth of allocations
// later, long after that slot was last handed out).
class AtomicArena {
std::byte* base_;
std::size_t mask_; // bytes - 1, bytes a power of two
std::atomic<std::size_t> next_{0};
public:
explicit AtomicArena(std::size_t bytes)
: base_(static_cast<std::byte*>(::operator new(bytes))), mask_(bytes - 1) {}
~AtomicArena() { ::operator delete(base_); }
void* allocate(std::size_t n) {
std::size_t o = next_.fetch_add(n, std::memory_order_relaxed); // the counter fix, again
return base_ + (o & mask_); // wrap, never out of bounds
}
};AtomicArena (fetch_add) : 0 slices handed to more than one thread
Measure
So there are three ways to hand out a block: plain new/delete, the shared arena bumping its cursor with fetch_add, and one private arena per thread. Here they are as the thread count climbs. The arenas never free a single block, they recycle the whole buffer at once when it fills, so they are timed the way they are actually used. The two tcmalloc columns come in a moment.
| threads | new / delete | tcmalloc (gperftools) | tcmalloc (Google) | shared arena | thread-local arena |
|---|---|---|---|---|---|
| 1 | 144 | 166 | 67.5 | 196 | 453 |
| 2 | 284 | 332 | 133 | 45.4 | 899 |
| 4 | 576 | 636 | 256 | 22.6 | 1,781 |
| 8 | 1,080 | 1,082 | 446 | 23.1 | 2,994 |
| 16 | 1,211 | 1,669 | 731 | 29.4 | 4,356 |
| 24 | 1,625 | 2,381 | 940 | 38.8 | 6,305 |
npm run bench:alloc (captured 2026-06-02).
tcmalloc (gperftools) is the identical binary run under
LD_PRELOAD=/lib64/libtcmalloc.so.4. tcmalloc (Google) is the same source statically linked against the newer google/tcmalloc via Bazel (it ships no preloadable library). The shared arena removed the lock but not the sharing. Every allocation is a fetch_add on the one cursor, a single cache line every core fights over, so its throughput collapses the moment more than one thread is bumping it, dropping below plain new. Lock-free does not mean contention-free.
The fastest lock is the one you never take
The arena that scales is the one where threads share nothing. Give each thread its own buffer and there is no lock, no atomic, and no shared cache line to fight over.
thread_local Arena tls_arena{ARENA_BYTES}; // one private arena per thread
void* tls_alloc() {
void* p = tls_arena.allocate(BLOCK);
if (!p) { tls_arena.reset(); p = tls_arena.allocate(BLOCK); } // bulk free, then retry
return p;
}Its throughput keeps climbing with the thread count, close to linear until the threads fill the machine and the rise tapers, while the shared arena collapses. The one cost is ownership. Each thread resets only its own buffer, so a slice handed out on one thread cannot be used by another. That fits work where a thread allocates and finishes with its own objects, and rules out handing them off.
What to use instead
The first two columns are the real lesson. Plain new/delete already scales, because a modern malloc keeps a separate cache per thread and only synchronizes when one thread frees another’s memory. The tcmalloc (gperftools) column makes it louder. It is the exact same binary, with the gperftools build of tcmalloc swapped in at load time, nothing recompiled.
# point LD_PRELOAD at libtcmalloc (find yours with: ldconfig -p | grep libtcmalloc)
LD_PRELOAD=/usr/lib64/libtcmalloc.so.4 ./alloc_benchIt tracks the system allocator and pulls clearly ahead at the higher thread counts. It still does not catch the thread-local arena, and that is the whole point: tcmalloc is solving the hard problem, freeing any block in any order and staying safe to share, while the arena gives all of that up. The way it scales is the same trick the arena uses, a per-thread cache: each thread serves most allocations from its own free list and only takes a lock on a shared central heap when that cache misses, which is why more threads keep helping instead of hurting.
The third column is the one that should win and does not. Google’s own tcmalloc, the newer rewrite, is a different library: Bazel-only and statically linked rather than preloaded, so the post builds the same source against it. It runs at about half the speed of the others, and not for any reason you would guess. It was not the build, which is -O3 with assertions off. It was not the per-CPU caches it is famous for, since turning them off changed nothing. A profile put the time somewhere more interesting: every allocation and free runs through a sampling-and-hooks dispatch, FreeWithHooksOrPerThread and alloc_small_sampled_hooks_or_perthread, the always-on machinery behind tcmalloc’s heap profiler. It spends a few nanoseconds per call deciding whether to record the allocation, even with sampling turned off. On a real server that buys continuous heap profiling for almost nothing. On a loop that does nothing but allocate and free, it is pure tax. The fancier allocator is slower here precisely because it is doing more, and what it is doing has nothing to do with this benchmark. Measure your own workload before you assume the sophisticated one wins.
That is why a general-purpose custom allocator is a fight you lose. The moment it has to free any block in any order and stay safe to share, it is doing tcmalloc’s and jemalloc’s job, and they have spent years on exactly that. The one that wins is narrow: give up freeing objects one at a time, or give up sharing across threads, and you do so little per call that nothing general-purpose can catch you.
So the takeaway is small. Run the harness with ThreadSanitizer to find the bug, and for general allocation measure first, then reach for tcmalloc or jemalloc before writing your own.