The ladder in nanoseconds
A CPU looks in L1 first, then L2, then L3, and only then goes out to RAM. Each step down is bigger and slower. L1 sits closest to the core, and L3 is shared by the cores. Data moves between the levels in blocks called cache lines. The Throughput Visualizer's reference ranges look like this:
- L1: 1 ns
- L2: 3 to 4 ns
- L3: 10 to 15 ns
- RAM: 50 to 80 ns
Real chips vary with the design and the clock, but the shape holds: each level costs several times the one above it.
What a miss costs
Nanoseconds hide how bad a miss is, because a core runs billions of cycles a second. At 4 GHz an L3 hit takes 40 to 60 cycles, and a trip to RAM takes 200 to 320. Fastest to fastest, RAM takes 5 times as long as L3. Those are cycles the core spends waiting unless it has other work it can start.
Why more cache helps some programs and not others
What matters is how much data a program touches again and again, its working set. While the working set fits in a cache level, the program runs at that level's speed. Once it spills past L3, most accesses go to RAM and the program slows to RAM's pace. The L3 cache vs RAM page shows that cliff.
Programs that revisit the same data, like databases and simulations, benefit the most from a bigger cache. Programs that stream through a huge file once gain little, since nothing gets reused. A cache size on a spec sheet is one number among many, so compare chips from the same family and check which tier it adds to.