Why 64 KB wasn't enough
The TCP header gives the receive window 16 bits, so the largest value it can carry is 65,535 bytes. On a long path that's a hard ceiling. At 100 ms, one connection tops out at 5.2 Mbps, however fast the link is.
Window scaling is an option that multiplies the 16 bit value by a power of two. The shift count goes up to 14, which allows a window just under 1 GiB. RFC 7323 defines it and replaces RFC 1323, the document older tools still cite.
How the two sides agree
The option travels only in the SYN segments that open the connection, and both sides must send it. If either SYN leaves it out, both ends set their shift to zero and the connection runs with the plain 64 KB window for its whole life. The shift is fixed once the connection opens, and each direction has its own.
That's the weak spot. Anything that rewrites or drops the option on the way, such as a misconfigured firewall, leaves a long distance transfer stuck at a flat low speed with no error message.
What it changes on a real path
On a 1 Gbps link, here's the ceiling without scaling and the window that fills the link:
- 20 ms: 26.2 Mbps unscaled, needs 2,442 KiB
- 50 ms: 10.5 Mbps unscaled, needs 6,104 KiB
- 100 ms: 5.2 Mbps unscaled, needs 12,208 KiB
- 200 ms: 2.6 Mbps unscaled, needs 24,415 KiB
With scaling on and a 16 MiB window, the window alone would allow 1.34 Gbps at 100 ms, past what the 1 Gbps link carries. The link is the limit again.
Checking that it's on
On Linux, net.ipv4.tcp_window_scaling defaults to 1, and tcp_moderate_rcvbuf is also on, so the receive buffer grows to match the path up to the top value of tcp_rmem. You rarely need to switch scaling on. You check that nothing blocks it and that the buffer limits cover the bandwidth-delay product of your path.
The giveaway is a transfer that sits at the same low speed whatever else you change. A flat 5.2 Mbps on a 100 ms path is the 64 KB window talking.