Encodings and Unicode

UTF-8 binaries

Modern Erlang keeps text in UTF-8 binaries at the edges of a program. A binary literal with non-ASCII characters holds the UTF-8 bytes of each one.

String functions from the string module understand Unicode and count characters, not bytes. The two numbers below are the whole lesson in one place.

erl

1> <<"café"/utf8>>.
<<"café"/utf8>>
2> string:length(<<"café"/utf8>>).
4
3> byte_size(<<"café"/utf8>>).
5

Latin-1 and the old days

Old Erlang printed charlists as Latin-1 by default, which is why legacy code mentions latin1. A charlist still prints as text when every code point is printable, so the shell answer below reads café and not a list of numbers.

The numbers are the code points. Ask for them with the tilde w directive when the encoding matters, and note that this word fits in Latin-1: its four code points are also its four Latin-1 bytes, while UTF-8 needs five bytes.

erl

1> unicode:characters_to_list(<<"café"/utf8>>).
"café"
2> io:format("~w~n", [unicode:characters_to_list(<<"café"/utf8>>)]).
[99,97,102,233]
ok

The unicode module

The unicode module converts between binaries, charlists, and UTF-16 or UTF-32. Its functions answer either the converted data or an error tuple.

Convert once at the boundary, then work in your normal representation inside. This module wraps the two everyday conversions; save it as text.erl.

-module(text). -export([upper/1, codepoints/1, from_codepoints/1]). upper(Bin) -> string:uppercase(Bin). codepoints(Bin) -> unicode:characters_to_list(Bin). from_codepoints(List) -> unicode:characters_to_binary(List).

erlc text.erl
erl

1> text:from_codepoints([99, 97, 102, 233]).
<<"café"/utf8>>
2> unicode:characters_to_nfc_binary(<<"cafe", 769/utf8>>).
<<"café"/utf8>>

The second call is normalization: e followed by a combining accent becomes the single code point for é. Compare text before normalizing, or the same word compares unequal to itself.

Rules of thumb

Keep binaries for data in and out, and only convert to charlists when an old API demands one. Never mix encodings inside one program.

When text looks broken, check the encoding at the boundary before blaming the logic.

erl

1> text:upper(<<"café"/utf8>>).
<<"CAFÉ"/utf8>>

Next: Time and date

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. How many bytes does <<"café"/utf8>> hold?

    Answer

    5

    The é needs two UTF-8 bytes, so four characters take five bytes.

  2. What does string:length(<<"café"/utf8>>) answer?

    Answer

    4

    The string module counts characters, and café has four of them.

  3. Why do string:length/1 and byte_size/1 disagree here?

    Answer

    Length counts characters, byte_size counts bytes

    One non-ASCII character takes several bytes, so characters and bytes are different numbers.

  4. What does unicode:characters_to_list(<<"café"/utf8>>) answer?

    Answer

    "café"

    It answers a charlist, and a printable charlist is printed as text: "café".

  5. How do you see the code points of that list?

    Answer

    Print it with ~w

    ~w prints the raw terms, so the list appears as [99,97,102,233]; ~p prints it as text.

  6. What does unicode:characters_to_binary([99, 97, 102, 233]) answer?

    Answer

    <<"café"/utf8>>

    The list holds code points, so the binary is their UTF-8 encoding, which the shell prints as <<"café"/utf8>>.

  7. What is unicode:characters_to_nfc_binary(<<"cafe", 769/utf8>>) for?

    Answer

    Normalizing e plus a combining accent into one code point

    NFC composition turns the two code points into the single é, so equal words compare equal.

  8. What does text:upper(<<"café"/utf8>>) answer?

    Answer

    <<"CAFÉ"/utf8>>

    string:uppercase/1 keeps the UTF-8 binary shape and uppercases the characters.

  9. Where should conversions happen?

    Answer

    Once, at the boundary of the program

    Convert at the edges, then work in one representation inside; mixed encodings are the usual source of broken text.

  10. Text arrives as bytes but looks wrong. What do you check first?

    Answer

    The encoding at the boundary

    Broken text is nearly always a wrong or missing conversion at the edge, not a logic bug.

The examples on this page were compiled and run with Erlang/OTP 27 (erts-15.2.7.14) on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev