Sizing the pipe
Picture a link as a pipe. Its speed is how wide the pipe is, and the round trip time is how long it takes to see the first acknowledgment come back. To keep the pipe full, the sender has to keep a pipe's worth of data in flight before that first acknowledgment arrives. Link speed times round trip time is that amount.
For a 1 Gbps link:
- 0.2 ms across a LAN: 25 KB
- 10 ms across a country: 1.3 MB
- 50 ms to another continent: 6.3 MB
- 100 ms to a distant region: 12.5 MB
- 200 ms over a poor route: 25 MB
A 10 Gbps link at 100 ms needs 125 MB in flight.
The 64 KB trap
Classic TCP advertises a receive window of at most 65,535 bytes, so that's all the sender may leave unacknowledged. Throughput is then the window divided by the round trip time, whatever the link can carry.
At 100 ms, a 64 KB window moves 5.2 Mbps, which is 0.52% of a gigabit link. A 10 Gbps link at the same 100 ms gives the same 5.2 Mbps, because the window sets the speed. To fill the 1 Gbps link, the receiver needs a window of at least 12,208 KiB.
Getting the window big enough
TCP window scaling is the option that lets the window grow past 64 KB. On Linux it's on by default, and receive buffers autotune up to the third value of tcp_rmem, which defaults to somewhere between 131072 bytes and 32 MB depending on RAM. An app that sets SO_RCVBUF itself turns autotuning off for that socket.
The sending side has a limit too. The top value of tcp_wmem defaults to between 64 KB and 4 MB depending on RAM, so a path that needs 12.5 MB in flight can need those limits raised.
A second fix is to open several connections. Each one has its own window, so eight connections carry eight windows' worth. That's why multi stream copy tools hold up better on long routes.