Encodings and Unicode
UTF-8 binaries
Modern Erlang keeps text in UTF-8 binaries at the edges of a program. A binary literal with non-ASCII characters holds the UTF-8 bytes of each one.
String functions from the string module understand Unicode and count characters, not bytes. The two numbers below are the whole lesson in one place.
erl
1> <<"café"/utf8>>.
<<"café"/utf8>>
2> string:length(<<"café"/utf8>>).
4
3> byte_size(<<"café"/utf8>>).
5
Latin-1 and the old days
Old Erlang printed charlists as Latin-1 by default, which is why legacy code mentions latin1. A charlist still prints as text when every code point is printable, so the shell answer below reads café and not a list of numbers.
The numbers are the code points. Ask for them with the tilde w directive when the encoding matters, and note that this word fits in Latin-1: its four code points are also its four Latin-1 bytes, while UTF-8 needs five bytes.
erl
1> unicode:characters_to_list(<<"café"/utf8>>).
"café"
2> io:format("~w~n", [unicode:characters_to_list(<<"café"/utf8>>)]).
[99,97,102,233]
ok
The unicode module
The unicode module converts between binaries, charlists, and UTF-16 or UTF-32. Its functions answer either the converted data or an error tuple.
Convert once at the boundary, then work in your normal representation inside. This module wraps the two everyday conversions; save it as text.erl.
-module(text).
-export([upper/1, codepoints/1, from_codepoints/1]).
upper(Bin) ->
string:uppercase(Bin).
codepoints(Bin) ->
unicode:characters_to_list(Bin).
from_codepoints(List) ->
unicode:characters_to_binary(List).
erlc text.erl
erl
1> text:from_codepoints([99, 97, 102, 233]).
<<"café"/utf8>>
2> unicode:characters_to_nfc_binary(<<"cafe", 769/utf8>>).
<<"café"/utf8>>
The second call is normalization: e followed by a combining accent becomes the single code point for é. Compare text before normalizing, or the same word compares unequal to itself.
Rules of thumb
Keep binaries for data in and out, and only convert to charlists when an old API demands one. Never mix encodings inside one program.
When text looks broken, check the encoding at the boundary before blaming the logic.
erl
1> text:upper(<<"café"/utf8>>).
<<"CAFÉ"/utf8>>
Next: Time and date
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
How many bytes does
<<"café"/utf8>>hold?Answer
5
The é needs two UTF-8 bytes, so four characters take five bytes.
-
What does
string:length(<<"café"/utf8>>)answer?Answer
4
The string module counts characters, and café has four of them.
-
Why do
string:length/1andbyte_size/1disagree here?Answer
Length counts characters, byte_size counts bytes
One non-ASCII character takes several bytes, so characters and bytes are different numbers.
-
What does
unicode:characters_to_list(<<"café"/utf8>>)answer?Answer
"café"It answers a charlist, and a printable charlist is printed as text:
"café". -
How do you see the code points of that list?
Answer
Print it with
~w~wprints the raw terms, so the list appears as[99,97,102,233];~pprints it as text. -
What does
unicode:characters_to_binary([99, 97, 102, 233])answer?Answer
<<"café"/utf8>>The list holds code points, so the binary is their UTF-8 encoding, which the shell prints as
<<"café"/utf8>>. -
What is
unicode:characters_to_nfc_binary(<<"cafe", 769/utf8>>)for?Answer
Normalizing e plus a combining accent into one code point
NFC composition turns the two code points into the single é, so equal words compare equal.
-
What does
text:upper(<<"café"/utf8>>)answer?Answer
<<"CAFÉ"/utf8>>string:uppercase/1keeps the UTF-8 binary shape and uppercases the characters. -
Where should conversions happen?
Answer
Once, at the boundary of the program
Convert at the edges, then work in one representation inside; mixed encodings are the usual source of broken text.
-
Text arrives as bytes but looks wrong. What do you check first?
Answer
The encoding at the boundary
Broken text is nearly always a wrong or missing conversion at the edge, not a logic bug.
The examples on this page were compiled and run with Erlang/OTP 27 (erts-15.2.7.14) on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev