Mojibake: text in the wrong encoding, and how to read it back

Mojibake is text that shows the wrong characters because the bytes of one encoding were read as another. The file is usually not corrupt: it is valid UTF-8, every byte sequence in it is a legal character, and no validator complains. A human sees the damage, nothing else does.

The word is Japanese: 文字化け, "character transformation". The cause is an ordinary mismatch, not an exotic one: a page served without a charset, an editor that saves as Windows-1252, a database connection set to latin1, a mail header encoded twice. This page shows what each mismatch looks like, how to reproduce it on purpose, how to read it back, and how to keep tests that catch it.

Tools used here: Python 3.14, PHP 8.3, pytest 9.2, bash 5.3 and the iconv command of this container. Every block below was run, and the output under it is copied from the run.

What mojibake is

A character is a number, and an encoding is the rule that writes that number as bytes. UTF-8 writes é as two bytes, C3 A9. Windows-1252, the encoding of most Western European Windows programs, writes the same character as one byte, E9. Hand the two UTF-8 bytes to a cp1252 decoder and it reads two characters, Ã and ©, because that is what cp1252 says C3 and A9 mean.

text = "é" print(text, text.encode(), text.encode("cp1252")) print(text.encode(), "->", text.encode().decode("cp1252"))

é b'\xc3\xa9' b'\xe9' b'\xc3\xa9' -> é

Python writes UTF-8 when encode() gets no argument, so the examples here leave it out; encode("utf-8") is the same call spelled out.

Two things must happen for the damage to stick. The reader must accept the bytes: cp1252, cp1251, latin-1 and koi8-r map almost all 256 byte values to characters, so a line of ordinary text is read without an error and without a warning. And the result must be saved as text again, so the file or the database now holds the wrong characters in a valid encoding. That is why the damage survives every later check: nothing in the bytes is illegal.

Types of mojibake

The damage differs by which encoding did the reading and by what happened to the result afterwards. Eight forms cover almost every case in practice:

Forms of the damage
What happenedWhat you see
UTF-8 read as cp1252 or latin-1café becomes café
UTF-8 read as cp1251Привет becomes Привет
double encoding: the damaged text saved and read wrong againcafé becomes café
replacement characters: a decode with errors='replace'Привет becomes 12 times U+FFFD, and the bytes are gone
a lossy conversion: the character does not exist in the target encodingПривет becomes ??????
a BOM saved with the filean invisible U+FEFF before the first character
a truncated multi-byte sequencecafé cut after four bytes becomes caf and U+FFFD
mixed in one file: part rewritten, part notline 1 fine, line 2 broken, line 3 fine

The first two forms, one Latin and one Cyrillic, are what a reader meets most often. The same UTF-8 text also looks different depending on which single-byte encoding read it:

utf8 = "Привет".encode() print("Привет as cp1251:", utf8.decode("cp1251")) print("Привет as cp1252:", utf8.decode("cp1252")) print("café as cp1252 :", "café".encode().decode("cp1252")) print("café as latin-1 :", "café".encode().decode("latin-1")) print("a dash as cp1252:", "—".encode().decode("cp1252")) print("a dash as latin-1:", repr("—".encode().decode("latin-1")))

Привет as cp1251: Привет Привет as cp1252: Привет café as cp1252 : café café as latin-1 : café a dash as cp1252: — a dash as latin-1: 'â\x80\x94'

cp1252 and latin-1 agree on the range A0-FF and differ in 80-9F, where cp1252 has the euro sign, the dashes and the curly quotes and latin-1 has control characters. A damaged dash is — under cp1252 and a letter followed by two control characters under latin-1, which is how the two are told apart in a file.

The third form is the same read applied twice: the damaged text was saved as UTF-8 and read as cp1252 once more. Every round trip doubles the number of bytes:

once = "café".encode().decode("cp1252") twice = once.encode().decode("cp1252") print("once :", once, once.encode()) print("twice:", twice, twice.encode())

once : café b'caf\xc3\x83\xc2\xa9' twice: café b'caf\xc3\x83\xc6\x92\xc3\x82\xc2\xa9'

The remaining forms are one-way, except the BOM. A decode with errors="replace", a save to a smaller encoding and a cut in the middle of a character all throw away information, so no guess brings it back: a replacement character and a question mark say only that something was there.

raw = "Привет".encode() broken = raw.decode("ascii", errors="replace") print("bytes in :", len(raw)) print("as ascii :", broken[:6].encode("unicode_escape").decode()) print("U+FFFD count :", broken.count("\ufffd")) print("bytes out :", len(broken.encode())) print("ascii replace:", "Привет".encode("ascii", errors="replace").decode()) cut = "café".encode()[:4].decode(errors="replace") print("truncated :", cut.encode("unicode_escape").decode())

bytes in : 12 as ascii : \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd U+FFFD count : 12 bytes out : 36 ascii replace: ?????? truncated : caf\ufffd

Twelve bytes came in and thirty-six went out, and not one of the thirty-six carries a letter of the original. The last line is a file cut in the middle of é: the leading byte of the character is left without its continuation, and the decoder marks the hole.

A BOM is different: U+FEFF is a legal character, zero-width and invisible, and the byte order mark at the start of a UTF-8 file is even allowed. It becomes damage when the reader keeps it as text, when a second one appears after a second save, or when the file is a PHP page: everything before <?php is printed to the browser, and the headers have already been sent.

text = "\ufeffcafé" print("characters:", len(text)) print("bytes :", text.encode()) print("first char:", text[0].encode("unicode_escape").decode())

characters: 5 bytes : b'\xef\xbb\xbfcaf\xc3\xa9' first char: \ufeff

The mixed case is the hardest to notice: one file, several lines, some rewritten correctly and some damaged. The whole file is valid UTF-8, so iconv and every validator pass it, and only a reader sees the broken line. A repair attempt line by line finds it: a line that comes back as different text was damaged, a line that cannot be read as the candidate encoding was not.

lines = ["Первая строка", "Вторая строка", "Третья строка"] data = "\n".join(lines).encode() data.decode("utf-8") print("bytes:", len(data), "- no UnicodeDecodeError, the file is valid UTF-8") for number, line in enumerate(data.decode("utf-8").splitlines(), 1): try: fixed = line.encode("cp1251").decode("utf-8") except UnicodeError: fixed = line print(number, "ok " if fixed == line else "bad", fixed)

bytes: 104 - no UnicodeDecodeError, the file is valid UTF-8 1 ok Первая строка 2 bad Вторая строка 3 ok Третья строка

Making it on purpose

A test needs the damaged text on purpose: as a fixture, as the body of a bug report, as input for a validator. Do not type it by hand: nobody can read it, an editor may re-encode it on save, and a copy from a browser can add characters of its own. Generate it from the correct text with the pair of encodings you want to test, so the fixture says what it means.

One line in Python:

python3 -c "print('café'.encode().decode('cp1252'))"

café

As a function, with the reader as an argument, and with the double round trip as one more call:

def damage(text, reader="cp1252"): return text.encode().decode(reader) # encode() writes UTF-8 print(damage("café")) print(damage("Привет", reader="cp1251")) print(damage(damage("café")))

café Привет café

In PHP the wrong-encoding read is one call: mb_convert_encoding($text, "UTF-8", "Windows-1252") treats the bytes of $text as Windows-1252 and writes them out as UTF-8. The repair is its inverse: iconv("UTF-8", "Windows-1252", $broken) from the iconv() function turns the two visible characters back into the two bytes the string started with:

<?php $text = 'café'; // UTF-8: 63 61 66 C3 A9 $broken = mb_convert_encoding($text, 'UTF-8', 'Windows-1252'); echo $broken, "\n"; echo iconv('UTF-8', 'Windows-1252', $broken), "\n";

Saved as mojibake.php, the script prints the damage and then the repair:

php mojibake.php

café café

In a shell the same pair is two calls of iconv: read the input as the encoding that never fails, write it as UTF-8. The printf escapes stand for the two bytes C3 A9 of é, which is the point of the example: the input is bytes, not characters.

printf 'caf\xc3\xa9\n' | iconv -f CP1252 -t UTF-8

café

To see the bytes rather than what the terminal makes of them, add xxd:

printf 'caf\xc3\xa9\n' | iconv -f CP1252 -t UTF-8 | xxd

00000000: 6361 66c3 83c2 a90a caf.....

The repair runs the other way, from UTF-8 to cp1252, and gives back the bytes the file started with:

printf 'caf\xc3\x83\xc2\xa9\n' | iconv -f UTF-8 -t CP1252

café

Reading it back

Damaged text announces itself with a few sequences that almost never appear in correct text, and the family they belong to tells which encoding did the reading:

Telltale sequences
You seeWhat read the bytes
Ã, Â, †before a letter or a punctuation mark: é, ü, —UTF-8 read as cp1252 or latin-1
à and  where a single damage has à and Â, and †before a curly quotethe same damage, a second time
Ð and Ñ before a high character: Пр, инUTF-8 Cyrillic read as cp1252
Р and С before џ, Ђ, ‚, µ: РџСЂ, РёРЅUTF-8 Cyrillic read as cp1251
U+FFFD, or an empty box in a terminala lossy decode: nothing to repair
a run of ?a lossy conversion: nothing to repair
an invisible character before the first lettera BOM

The repair is the same operation in reverse: encode the visible text with the candidate encoding, then decode the result as UTF-8. The right candidate gives the original back. A wrong one either cannot represent the characters at all, or produces bytes that are not valid UTF-8. When more than one candidate survives, count the letters of the language you expect: the repaired text should be almost all of them.

CYRILLIC = set("абвгдеёжзийклмнопрстуфхцчшщъыьэюяАБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ ") def score(text): return sum(ch in CYRILLIC for ch in text) / len(text) broken = "Привет" print(f"before: {score(broken):.2f}") for enc in ("cp1251", "cp1252", "latin-1", "koi8-r", "mac_cyrillic"): try: fixed = broken.encode(enc).decode("utf-8") except UnicodeError: print(f"{enc:13} cannot hold these characters") continue print(f"{enc:13} {score(fixed):.2f} {fixed}")

before: 0.58 cp1251 1.00 Привет cp1252 cannot hold these characters latin-1 cannot hold these characters koi8-r cannot hold these characters mac_cyrillic cannot hold these characters

Only cp1251 can hold the characters, and it repairs the line to Привет: the score of the damaged line was 0.58, the score of the repair is 1.00. The score is a heuristic: it ranks the candidates, and on a short string two of them can fit.

A file damaged twice needs two rounds and a file damaged once needs one, so the number of rounds tells them apart. Apply the repair until it stops working, and count how many times it changed the text:

def repair(text, enc): for attempt in range(1, 5): try: text = text.encode(enc).decode("utf-8") except UnicodeError: return attempt - 1, text return 4, text single = "café".encode().decode("cp1252") double = single.encode().decode("cp1252") for name, broken in (("single", single), ("double", double)): repairs, fixed = repair(broken, "cp1252") print(f"{name:7} rounds={repairs} {fixed}")

single rounds=1 café double rounds=2 café

A third round raises UnicodeDecodeError, because the repaired text is no longer a valid cp1252 reading of UTF-8 bytes. That is the stopping rule: repair as many times as the file needs and no more.

A repair is a guess, and the guess is better on long text than on short text. "é" could stand for one character read wrong or for two characters written that way on purpose. Before a repair is trusted, it is compared with a second copy: the version in the repository, the source table, the same text on another page. When the text already contains U+FFFD or question marks there is nothing to compare, and only the source can give the original back.

In tests

A test suite needs three things from mojibake: a fixture set that says which damage means which repair, a round-trip test that proves the pair of encodings is lossless, and a regression test that the code which detects damage still detects it.

The fixtures build their damaged side from the correct text at import time, so the test file itself holds no mojibake for an editor, a hook or a reader to trip over. Each case is a good string, the same string after the wrong read, and the encoding that undoes it:

import pytest CASES = [ ("café", "café".encode().decode("cp1252"), "cp1252"), ("Привет", "Привет".encode().decode("cp1251"), "cp1251"), ("café — 5 €", "café — 5 €".encode().decode("cp1252"), "cp1252"), ] PAIRS = ("é", "Ã", "©", "П", "Ñ€", "Рџ", "СЂ", "â€") def repair(text, enc): return text.encode(enc).decode("utf-8") def looks_broken(text): return any(pair in text for pair in PAIRS) @pytest.mark.parametrize(("good", "broken", "enc"), CASES) def test_repair_restores_the_text(good, broken, enc): assert broken != good assert repair(broken, enc) == good @pytest.mark.parametrize(("good", "broken", "enc"), CASES) def test_broken_form_is_flagged(good, broken, enc): del good, enc assert looks_broken(broken) def test_clean_text_is_not_flagged(): for text in ("café", "Привет", "Japanese text", "naïve — 5 €"): assert not looks_broken(text) def test_correct_pair_round_trips(): for text in ("café", "Привет", "日本語", "naïve — 5 €", ""): assert text.encode("utf-8").decode("utf-8") == text def test_lossy_pair_does_not_round_trip(): text = "Привет" assert text.encode("ascii", errors="replace").decode() != text

pytest -q tests/test_mojibake.py

......... [100%] 9 passed in 0.03s

The first two tests are one table read twice: the repair must return the good string, and the detector must flag the broken one. The detector looks for pairs, not for single characters, because Р and С are ordinary Russian letters; what marks damage is the pair, not either letter alone. The last two tests are the property itself: decode(encode(x)) == x holds for UTF-8 and fails for a lossy conversion, so a pair that claims to be lossless is proved by the data, not by its documentation.

In this repository git runs the same rule on every commit. The pre-commit check for line-ending and encoding damage counts the double-encoded sequences a staged change adds, with one pattern per family: the Ã/Â family, the Ð/Ñ family and the Р/С family. Ten new sequences block the commit with an ARTIFACT line and the count; one sample word, which is what a page that mentions mojibake in passing carries, stays under the limit. The check compares the new side of a file with its base side, so damage committed earlier never fails a later commit, and a translation, which rewrites every line, passes.

This page itself is over that limit: it quotes too many samples to pass, so a commit that adds it has to carry SKIP=check-diff-artifacts, the intended-hit route the hook page describes. The tests of the hook take the same route from the other side: they build the damaged content at run time, so the test file never carries the mojibake it tests.

Article author: Arthur Isaev

Banner image