Caching under load

The experiment is a discrete-event simulation. Instead of timers, it keeps a priority queue of future events (request arrivals, origin responses, completions) and processes them in time order on a simulated clock. Playback speed only controls how fast that clock advances on screen; the numbers are identical at any speed, and Step processes exactly one event.

How a request is processed

1. Requests arrive at random (a Poisson process) at the chosen rate. Each picks a key using a Zipf distribution: key k1 is the most popular, and the skew controls how much more popular.
2. The cache is checked first, costing 2 ms.
3. Hit (stored and not expired): answered in 2 ms. The entry becomes the most recently used. A hit does not extend its TTL.
4. Miss: the request goes to the origin and takes 2 ms + origin latency. Origin latency is your setting ±20%, sampled once per request as part of the workload.
5. When the origin responds, the value is stored. TTL starts at that moment, not when the request arrived.
6. If the cache is full, expired entries are purged first; if it is still full, the least recently used entry is evicted.
7. Expiry is lazy: an expired entry stays until it is looked up again or space is needed, which is why the inspector can show expired rows.

How the comparison is fair

The request stream (arrival times, keys and per-request origin latency) comes from a seeded random generator, and both runs consume the same stream. The no-cache run sends every request to the origin and its latency is exactly that request's origin latency. Metrics are counted from processed events: lookups when a request arrives, latency only when its response completes.

Under the hood

The simulation is plain TypeScript with no UI code: a seeded PRNG (Mulberry32), a binary-heap event queue and the cache model. It is covered by unit tests for hits, TTL timing, LRU order, coalescing, metric calculations and run-for-run repeatability. The React layer only advances the clock each animation frame and renders the current state.

Tradeoffs it shows

• Freshness vs load: a short TTL serves fresher data but sends more traffic to the origin.
• Capacity vs working set: once the set of regularly requested keys exceeds capacity, LRU starts evicting entries that are about to be needed.
• Skew matters more than size: with a few hot keys a small cache works well; with uniform traffic even a large one barely helps.
• Misses are slightly slower than no cache at all, because of the extra lookup. Caching pays off only above a certain hit rate.
• Stampedes: without coalescing, a popular key expiring sends a burst of identical requests to the origin.

What this model leaves out

• It is a simplified model for building intuition, not a production benchmark. Real numbers depend on your hardware, network and data.
• The origin has unlimited capacity: it never queues, slows down under load or fails.
• There are no writes, so no invalidation or stale reads; entries only leave by TTL or eviction.
• One cache node, fixed lookup cost, values of equal size and no network time between client and cache.
• Real caches use refinements this one doesn't: approximate LRU (e.g. Redis samples keys), LFU or segmented policies, stale-while-revalidate and jittered TTLs.