Encodings and Unicode
| UTF-8 binaries | |
| Latin-1 and the old days | |
| The unicode module | |
| Rules of thumb |
UTF-8 binaries
Modern Erlang stores text as UTF-8 binaries. A literal string with non-ASCII characters is a binary holding the UTF-8 bytes.
String functions from the string module understand Unicode and count characters, not bytes.
1> "caf" ++ <<233:8>>.
<<"caf"/utf8>>
2> string:length("caf" ++ <<233:8>>).
4
Latin-1 and the old days
Old Erlang printed charlists as Latin-1 by default, which is why legacy code mentions latin1. Binaries and lists still convert through the encoding modules.
New code picks one encoding, usually UTF-8, and converts at the edges only.
[99,97,102,233]
The unicode module converts between binaries, charlists, and UTF-16 or UTF-32. Its functions answer either the converted data or an error tuple.
2>
Keep binaries for data in and out, and only convert to charlists when an old API demands one. Never mix encodings inside one program.
Next:
Time and date
Article author: Arthur Isaev1> unicode:characters_to_list("caf" ++ <<233:8>>).
The unicode module
Convert once at the boundary, then work in your normal representation inside.
1> unicode:characters_to_binary(["caf", 233]).
<<"caf"/utf8>>unicode:characters_to_nfc_binary("cafe" << 769:8>>).
<<"caf"/utf8>>
Rules of thumb
When text looks broken, check the encoding at the boundary before blaming the logic.
1> upper(Bin) ->
string:uppercase(Bin).