What the spec number says
Drive data sheets give an unrecoverable read error rate as one error per so many bits read. A rate of 1 in 10^14 bits works out to one error per 12.5 TB read, and 1 in 10^15 bits is one per 125 TB. The error is a sector the drive can't return even after its own retries and error correction. It's one bad sector, not a dead drive, and the drive keeps running.
The rate depends on the drive line. WD lists its Red Plus NAS drives, 2 to 12 TB, at less than 1 in 10^14, and its Red Pro line at less than 1 in 10^15 on nearly every model. Seagate's IronWolf data sheet lists 1 per 10^14 for its 6 TB and smaller drives and 1 per 10^15 from 10 TB up, and IronWolf Pro drives list 1 per 10^15. Read the sheet for your exact model. The wording is a maximum, not a measurement.
What it does to a rebuild
A rebuild reads every bit on the surviving drives, so the data read grows with the array. If every bit failed on its own at exactly the spec rate, the chance of at least one error would be 1 minus (1 minus the rate) raised to the number of bits read. The RAID Calculator uses that formula. For a RAID 5 rebuild:
- Four 4 TB drives read 12 TB. That's at most about 62% at 1 in 10^14 and 9% at 1 in 10^15.
- Four 8 TB drives read 24 TB. That's 85% and 17%.
- Six 20 TB drives read 100 TB. Drives that size, like the IronWolf Pro 20 TB, are rated 1 in 10^15, which gives 55%.
A mirror reads one drive. Rebuilding a pair of 8 TB drives reads 8 TB, which gives at most about 47% at 1 in 10^14.
Why those numbers are ceilings
Those figures treat every bit as an independent coin flip at the spec rate. Real drives don't work like that. The spec is a maximum, so the real rate is at or below it. Errors also cluster. A 2007 study of 1.53 million drives in NetApp storage systems found that 3.45% developed latent sector errors over 32 months. Nearline SATA drives were hit far more often than enterprise ones, about 8.5% against 1.9%. For most models, over 80% of the drives with errors had fewer than 50 of them, and a drive with one error was more likely to develop more. The study also found that more than 60% of the errors turned up during scrubs.
So read the list above as a worst case on paper, not a forecast for your array. The real chance depends on the drive model and its age. It's a real risk all the same, because one bad sector with no parity left to cover it is data you can't reconstruct, and each extra terabyte read adds to the risk.
What happens when one hits
The outcome depends on the RAID level and the controller. In RAID 5, or in a two drive mirror that has already lost a drive, there's nothing left to rebuild that sector from. Some controllers mark the block unreadable and finish the rebuild, so you lose a file or a few. Others abort the rebuild, and you restore from backup. RAID 6, RAID-Z2 and a three way mirror still have a spare copy or parity, so a single bad sector gets rebuilt from it.
Four habits keep the odds low:
- Scrub on a schedule. A scrub reads every block while the array is healthy, so a bad sector is found and rewritten from parity or the mirror before a failed drive makes it unfixable. Look for a scrub or consistency check in your NAS or RAID software.
- Use double parity for big drives, so a bad sector during a one drive rebuild is covered.
- Check the data sheet and pick a drive rated at 1 in 10^15 when you have the choice.
- Keep a backup, because RAID can't repair a read error that lands when no redundancy is left.