Developer

UTF-8

The most common way to store Unicode text as bytes. It writes each code point as one to four bytes, and plain ASCII text stays one byte per character, so any ASCII file is already valid UTF-8.

Bytes per character

The first byte of a UTF-8 character says how long it is, and every byte after it starts with the bits 10. A lead byte starting 0 is a one-byte character, 110 starts a two-byte one, 1110 a three-byte one and 11110 a four-byte one. The code point's own bits fill the rest. That gives four sizes:

  • U+0000 to U+007F: 1 byte. This is ASCII, so A is 41.
  • U+0080 to U+07FF: 2 bytes. Accented Latin, Greek, Cyrillic, Hebrew and Arabic. é is C3 A9.
  • U+0800 to U+FFFF: 3 bytes. Most Chinese, Japanese and Korean characters, such as 中, which is E4 B8 AD, and many symbols.
  • U+10000 to U+10FFFF: 4 bytes. Emoji and rare scripts. 😀 is F0 9F 98 80.

What that does to sizes

English text costs one byte per character, the same as ASCII. Russian or Greek text costs up to two bytes per letter, and Chinese, Japanese and Korean text usually costs three bytes per character. So a string's byte length usually isn't its character count, and a limit in bytes, like a database column or an API field, fills up sooner for some languages. The Character Counter counts visible characters or code points and shows the UTF-16 units and UTF-8 bytes of your text.

Base64 and URL encoding work on these bytes, not on characters. é is the two bytes C3 A9, so Base64 turns it into 4 characters, w6k=, and URL encoding writes it as %C3%A9.

Why UTF-8 won

Three design choices explain it. ASCII bytes stay ASCII: RFC 3629 notes that a plain ASCII string is also a valid UTF-8 string, and that ASCII byte values never appear inside a longer character. A parser hunting for /, , or a line feed can keep scanning bytes. UTF-8 is also self-synchronizing, since continuation bytes always start with 10, so a decoder can find the next character after a bad byte. And it has no byte order problem, because it's made of single bytes. A byte order mark (EF BB BF) can still show up at the start of a file, but RFC 3629 notes that its byte order job is useless in UTF-8, so it can only work as a signature.

The result is that about 99% of websites with a known encoding use UTF-8, according to W3Techs. JSON requires it for text exchanged between systems, and the HTML standard requires it for web pages.

Where it goes wrong

Bytes that aren't valid UTF-8, like a lone Latin-1 E9, can't be decoded, and software usually shows the replacement character � (U+FFFD) instead. Decoders also have to refuse overlong forms, like C0 80 for the null character, and the surrogate range U+D800 to U+DFFF.

The other classic failure is reading UTF-8 with the wrong encoding, which gives mojibake like é for é. Save files as UTF-8, and declare it in the charset=utf-8 part of the Content-Type header and with <meta charset="utf-8">.

Work it out yourselfOpen the Text Encoding Converter

Sources

Back to the glossary