A pre-commit check for line-ending and encoding damage: check-diff-artifacts
check-diff-artifacts.sh is a pre-commit hook that reads the staged diff and refuses one kind of change: damage at the level of bytes that scripted edits leave behind. A Python script that rewrites a page in text mode drops its CRLF line endings, an editor adds a BOM, a file read in the wrong encoding comes back as mojibake: garbled text, the bytes of one encoding shown as characters of another, so that café turns into café. None of these is ever intended, so the hook can block them without exceptions. Everything else passes, including whitespace-only edits and rewrites that change every line, such as a translation or a regenerated sitemap. A file whose only change is a conversion from CRLF to LF passes too, so a repository can move all its files to LF.
This page sets the hook up in a scratch repository and runs every rule with real output. Tools used here: git 2.43.0, bash 5.2, GNU grep 3.11, iconv from glibc 2.39, the pre-commit framework 4.6.2 and Python 3.12 for the edits that do the damage. How this version replaced the first one, and how it was tested against a whole repository history, is in Reworking a pre-commit check so it fires only on real damage. The first version and its git diff -w bug are in A pre-commit check that blocked a correct commit.
What it blocks and what it lets through
Every rule compares the new side of a file with its base side: the HEAD version for a staged change, the parent for a commit. Damage that is already in the base never fails a later commit. A finding that blocks is printed as ARTIFACT and the script exits with 1.
| Rule | Typical cause |
|---|---|
| CRLF lost in a CRLF file together with other changes | a Python or sed rewrite in text mode that also edits the file |
| a file with one ending style left with mixed endings | a line appended with echo to a CRLF file, a CRLF snippet pasted into an LF file |
| an LF file flipped to CRLF | an editor set to Windows line endings |
| U+FFFD replacement characters added to an existing file | a decode with errors='replace' |
| mojibake: 10 or more new double-encoded sequences | UTF-8 read as cp1252, latin-1 or cp1251 and saved again |
| a valid UTF-8 file no longer valid UTF-8 | a page saved as cp1251 or latin-1 |
| a UTF-8 BOM added to an existing file, or a new PHP file with a BOM | an editor set to "UTF-8 with BOM"; PHP prints the BOM before any header |
A finding that does not block is printed as NOTE, and the exit status stays 0:
| Change | Report |
|---|---|
| a conversion from CRLF to LF and nothing else, also as part of a move | NOTE |
| whitespace only: indentation, trailing spaces, a run of spaces | NOTE |
| a 1:1 rewrite of many lines: translation, a path renamed on every line, a regenerated file | nothing |
| a BOM removed | NOTE |
| stray CRs removed from a mostly-LF file | NOTE |
| a new file with mixed endings, U+FFFD or (not PHP) a BOM: there is no base to damage | NOTE |
| untracked, deleted, binary, symlink and submodule entries | skipped |
Setting it up
The demo runs in a new repository. The script itself is at the end of this page; save it as tools/check-diff-artifacts.sh:
git init -q demo && cd demo
git config user.email you@example.com && git config user.name You
mkdir -p tools
vi tools/check-diff-artifacts.sh
chmod +x tools/check-diff-artifacts.sh
The hook is a local entry of the pre-commit framework. It reads the whole staged diff itself, so it runs on every commit and takes no file names:
vi .pre-commit-config.yaml
repos: - repo: local hooks: - id: check-diff-artifacts name: check-diff-artifacts (staged) entry: tools/check-diff-artifacts.sh --staged language: system always_run: true pass_filenames: false
pre-commit install
pre-commit installed at .git/hooks/pre-commit
Four sample pages: one with CRLF endings, as a file first saved on Windows has, one of 120 lines, one in Russian and a short list:
printf '<?php\r\n// A page first saved on Windows\r\n$title = "Old title";\r\necho $title;\r\n' > legacy.php
for i in $(seq 1 120); do echo "<p>Draft note $i</p>"; done > notes.html
mkdir -p ru && printf '<h1>О проекте</h1>\n<p>Привет! Это страница о проекте, она написана по-русски.</p>\n' > ru/about.html
printf '<ul>\n <li>one</li>\n <li>two</li>\n</ul>\n' > list.html
git add . && git commit -qm "Adds the pages"
check-diff-artifacts (staged)............................................Passed
git ls-files --eol shows the ending style git sees in the index (i/) and in the working tree (w/):
git ls-files --eol
i/lf w/lf attr/ .pre-commit-config.yaml i/crlf w/crlf attr/ legacy.php i/lf w/lf attr/ list.html i/lf w/lf attr/ notes.html i/lf w/lf attr/ ru/about.html i/lf w/lf attr/ tools/check-diff-artifacts.sh
Line endings
The most common damage: a script changes one line of a CRLF page and writes the whole file back with LF. Python does this by default, because a file opened in text mode turns every \r\n into \n on reading and writes \n back on Linux:
python3 -c "p = 'legacy.php'; t = open(p).read(); open(p, 'w').write(t.replace('Old title', 'New title'))"
git add legacy.php
git commit -qm "Renames the page title"
check-diff-artifacts (staged)............................................Failed - hook id: check-diff-artifacts - exit code: 1 ARTIFACT: legacy.php [staged]: CRLF line endings lost (base CRLF on all 4 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
The last line points to the documentation of the repository the script comes from; change that path in the script for yours. The commit did not happen. The report also says what would pass: the same file converted to LF without the title change, which is how a file moves to LF. Here the title change is the point, so the repair puts the carriage returns back. Opened with newline='', Python reads and writes the line endings as they are, which is also how the edit should have been done in the first place:
python3 -c "p = 'legacy.php'; t = open(p, newline='').read(); open(p, 'w', newline='').write(t.replace('\n', '\r\n'))"
git add legacy.php
git diff --cached --numstat
1 1 legacy.php
One line changed instead of four: only the title. The commit goes through:
git commit -qm "Renames the page title"
check-diff-artifacts (staged)............................................Passed
Losing only some of the endings is caught as well. A line appended with echo ends in LF, and the CRLF file becomes mixed:
echo '<p>Added later</p>' >> legacy.php
git add legacy.php
tools/check-diff-artifacts.sh --staged
ARTIFACT: legacy.php [staged]: mixed line endings, 1 of 5 lines now end in LF in a CRLF file check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
sed gives every line exactly one CR. The \r* keeps it from doubling the CR on lines that already have one:
sed -i 's/\r*$/\r/' legacy.php
git add legacy.php
tools/check-diff-artifacts.sh --staged
check-diff-artifacts: clean
git commit -qm "Adds a line to the old page"
check-diff-artifacts (staged)............................................Passed
A move that flips line endings
A move is where damage hides best: the old path disappears and the file looks new. The script follows git's rename detection and compares the moved file with its source. Here the page moves into pages/, gets one line edited and loses its CRLF on the way:
mkdir -p pages && git mv legacy.php pages/legacy.php
sed -i 's/\r$//; s/Added later/Added later, moved/' pages/legacy.php && git add pages/legacy.php
git status --short
R legacy.php -> pages/legacy.php
tools/check-diff-artifacts.sh --staged
ARTIFACT: pages/legacy.php (from legacy.php) [staged]: CRLF line endings lost (base CRLF on all 5 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
sed -i 's/\r*$/\r/' pages/legacy.php && git add pages/legacy.php
tools/check-diff-artifacts.sh --staged
check-diff-artifacts: clean
git commit -qm "Moves the old page to pages/"
check-diff-artifacts (staged)............................................Passed
A move that only converts the file to LF, with no edit, passes with a note, like any other conversion. Git pairs a deleted and an added file as a rename when they are at least 50% similar. A move that also rewrites most of the file shows up as a new file, and a new file has no base to compare with: see Limits.
Encoding damage
Mojibake. A script that opens a UTF-8 file as cp1251 reads every Russian letter as two Windows characters and saves them as UTF-8. The page still is valid UTF-8, but the text is gone:
python3 -c "p = 'ru/about.html'; t = open(p, encoding='cp1251').read(); open(p, 'w', encoding='utf-8').write(t.replace('<h1>', '<h1 id=\"top\">'))"
head -n 2 ru/about.html
<h1 id="top">Рћ проекте</h1> <p>Привет! Рто страница Рѕ проекте, РѕРЅР° написана РїРѕ-СЂСѓСЃСЃРєРё.</p>
git add ru/about.html
tools/check-diff-artifacts.sh --staged
ARTIFACT: ru/about.html [staged]: mojibake introduced (0 -> 42 double-encoded sequences: UTF-8 read as cp1252, latin-1 or cp1251 and saved again) check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
The safest repair of encoding damage is the committed version: restore it and redo the edit with the right encoding. git restore --staged --worktree resets both the index and the file:
git restore --staged --worktree ru/about.html
A file saved in another encoding. iconv turns the page into cp1251, as an old editor would:
iconv -f utf-8 -t cp1251 ru/about.html > ru/about.tmp && mv ru/about.tmp ru/about.html
git add ru/about.html
tools/check-diff-artifacts.sh --staged
ARTIFACT: ru/about.html [staged]: no longer valid UTF-8 (illegal input sequence at position 4), saved in another encoding? check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
git restore --staged --worktree ru/about.html
A BOM. The three bytes EF BB BF at the start of a PHP file are sent to the browser before any header the page sets:
printf '\xef\xbb\xbf' | cat - pages/legacy.php > legacy.tmp && mv legacy.tmp pages/legacy.php
git add pages/legacy.php
tools/check-diff-artifacts.sh --staged
ARTIFACT: pages/legacy.php [staged]: UTF-8 BOM added (PHP prints it before any header) check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
git restore --staged --worktree pages/legacy.php
What passes
A whitespace-only edit is reported, so it does not go unnoticed, but it is a change someone made on purpose and the commit goes through. Here the list is re-indented with two spaces:
sed -i 's/^ / /' list.html
git add list.html
tools/check-diff-artifacts.sh --staged
NOTE: list.html [staged]: whitespace-only change (2+/2-: indentation, trailing spaces), not blocking check-diff-artifacts: clean (1 note(s), not blocking)
git commit -qm "Indents the list with two spaces"
check-diff-artifacts (staged)............................................Passed
The framework shows a hook's output only when the hook fails, so the note is visible when the script is run by hand. A rewrite that changes every line of a file, as many lines added as deleted, is just as ordinary. A translation done line by line or a sitemap with new dates looks the same as this:
sed -i 's/Draft/Final/' notes.html
git add notes.html
git diff --cached --numstat
120 120 notes.html
tools/check-diff-artifacts.sh --staged
check-diff-artifacts: clean
git commit -qm "Marks the notes final"
check-diff-artifacts (staged)............................................Passed
Checking without committing
The script takes one option that says which two sides it compares:
| Command | Base | New side |
|---|---|---|
| check-diff-artifacts.sh | index | working tree: unstaged changes |
| check-diff-artifacts.sh --staged | HEAD | index: what the next commit records |
| check-diff-artifacts.sh --all | HEAD | working tree: staged and unstaged together |
| check-diff-artifacts.sh --commit REV | first parent of REV | REV, a commit that is already made |
Without an option the script checks an edit before it is staged:
python3 -c "p = 'pages/legacy.php'; t = open(p).read(); open(p, 'w').write(t.replace('New title', 'Newer title'))"
tools/check-diff-artifacts.sh
ARTIFACT: pages/legacy.php [worktree]: CRLF line endings lost (base CRLF on all 5 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
git commit --no-verify skips every hook, so the damage gets into the history. --commit finds it there:
git commit --no-verify -qam "Renames the page title again"
tools/check-diff-artifacts.sh --commit HEAD
ARTIFACT: pages/legacy.php [commit b3b09761b]: CRLF line endings lost (base CRLF on all 5 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
A new commit that restores the CRLF would itself be an LF to CRLF flip, and the hook blocks it. While the commit is not pushed, take it apart instead. git reset --soft HEAD~1 removes the commit and keeps its changes staged. The base is then the good version again:
git reset --soft HEAD~1
sed -i 's/\r*$/\r/' pages/legacy.php && git add pages/legacy.php
tools/check-diff-artifacts.sh --staged
check-diff-artifacts: clean
git commit -qm "Renames the page title again"
check-diff-artifacts (staged)............................................Passed
When the damaged commit is already pushed, the edit is in the history and the file is LF. LF is fine: the next section moves files to LF on purpose. Only a file that must keep CRLF needs a repair, and that repair is an intended hit.
Moving files to LF
A file whose only change is the conversion from CRLF to LF is not damage: there is no edit to hide. The hook checks that the new file is exactly the base with the carriage return before each line break removed, and lets it through with a note. git ls-files --eol lists the files that are CRLF in the index:
git ls-files --eol | awk '$1 == "i/crlf" {print $NF}'
pages/legacy.php
sed -i 's/\r$//' pages/legacy.php && git add pages/legacy.php
git diff --cached --numstat
5 5 pages/legacy.php
tools/check-diff-artifacts.sh --staged
NOTE: pages/legacy.php [staged]: converted from CRLF to LF, nothing else changed (5 lines) check-diff-artifacts: clean (1 note(s), not blocking)
git commit -qm "Converts pages/legacy.php to LF"
check-diff-artifacts (staged)............................................Passed
git ls-files --eol pages/legacy.php
i/lf w/lf attr/ pages/legacy.php
Every line changed, and that is fine, because nothing else did. Convert in commits that only convert, by folder if the repository is large, and edit the files in other commits: a conversion and an edit of the same file in one commit is what the first example blocked.
Once everything is LF, a .gitattributes file keeps it so. text=auto lets git decide which files are text, and eol=lf stores them with LF, whatever an editor wrote:
printf '* text=auto eol=lf\n' > .gitattributes && git add .gitattributes && git commit -qm "Keeps text files LF"
check-diff-artifacts (staged)............................................Passed
printf 'one\r\ntwo\r\n' > win.txt && git add win.txt
warning: in the working copy of 'win.txt', CRLF will be replaced by LF the next time Git touches it
git ls-files --eol win.txt
i/lf w/crlf attr/text=auto eol=lf win.txt
The index holds LF (i/lf) although the file on disk has CRLF (w/crlf), so the line-ending rules of the hook have nothing left to find in the index; the encoding rules keep working. A file that must have CRLF on disk, such as a Windows batch file, gets its own line. Git still stores it with LF and writes CRLF when it checks the file out:
printf '*.bat text eol=crlf\n' >> .gitattributes && git add .gitattributes && git commit -qm "Keeps batch files CRLF in the working tree"
check-diff-artifacts (staged)............................................Passed
printf '@echo off\r\necho Building\r\n' > run.bat && git add run.bat
git ls-files --eol run.bat
i/lf w/crlf attr/text eol=crlf run.bat
git commit -qm "Adds run.bat"
check-diff-artifacts (staged)............................................Passed
An intended hit
A conversion to LF needs no skip. Few hits are still wanted: a repair to CRLF after a pushed damage, or a page that shows mojibake on purpose. Here the Russian page gets a sample of a wrong decode, built at run time from proper text, and the hook blocks the commit as usual:
python3 -c "p = 'ru/about.html'; s = 'Привет! Это страница о проекте.'.encode().decode('cp1251'); open(p, 'a', encoding='utf-8').write('<p>A wrong decode looks like this: ' + s + '</p>\n')"
git add ru/about.html
git commit -qm "Shows a wrong decode on the about page"
check-diff-artifacts (staged)............................................Failed - hook id: check-diff-artifacts - exit code: 1 ARTIFACT: ru/about.html [staged]: mojibake introduced (0 -> 21 double-encoded sequences: UTF-8 read as cp1252, latin-1 or cp1251 and saved again) check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
The framework's SKIP variable skips this one hook and runs all the others. The second -m says in the commit why the hook was skipped:
SKIP=check-diff-artifacts git commit -qm "Shows a wrong decode on the about page" -m "Committed with SKIP=check-diff-artifacts: the mojibake sample is intended."
check-diff-artifacts (staged)...........................................Skipped
git commit --no-verify would also work and would skip every other check with it. Keep it for a broken hook.
How the checks work
Which files and which sides. One git diff --raw call lists the changed files with the blob ids of both sides. -z keeps paths unquoted, so a name with Cyrillic letters or spaces reaches the checks as it is. -M pairs renames, and --no-abbrev prints full ids. A blob id of zeros means "the file in the working tree":
while IFS= read -r -d '' meta; do IFS= read -r -d '' path || break src="" read -r smode dmode ssha dsha status <<< "${meta#:}" case "$status" in R*|C*) src="$path"; IFS= read -r -d '' path || break ;; esac ... check_file "$path" "$src" "$ssha" "$dsha" done < <("${GIT_DIFF[@]}" --raw --no-abbrev "${RANGE[@]}")
The ... stands for the skip of symlinks, submodules and pure renames. Each side is written to a temporary file with git cat-file blob <id>. A file that git's --numstat reports as binary, with - instead of line counts, is skipped.
Line endings. For each side the script counts the lines (wc -l) and the lines that end in a CR (grep -c $'\r$'), and names the style: lf when no line has a CR, crlf when every line has one, mixed otherwise. The verdict is a pair of styles:
| Base | New | Report |
|---|---|---|
| crlf | lf | NOTE when only the CRs were removed, otherwise ARTIFACT: CRLF line endings lost |
| crlf | mixed | ARTIFACT: mixed line endings |
| lf | crlf | ARTIFACT: flipped from LF to CRLF |
| lf | mixed | ARTIFACT: mixed line endings |
| mixed, half or more CRLF | lf | NOTE when only the CRs were removed, otherwise ARTIFACT: CRLF line endings lost |
| mixed, mostly LF | lf | NOTE: stray CRs removed |
| none (new file) | mixed | NOTE |
| same style on both sides | nothing |
"Only the CRs were removed" is one exact comparison: the base with sed stripping the CR before each line break must be byte for byte the new file. One changed space anywhere fails it:
only_cr_removed() { LC_ALL=C sed 's/\r$//' "$1" | cmp -s - "$2"; }
The test is one way. LF to CRLF always blocks, because the point is a repository that ends up LF. When neither side has a single CR, the line count is not needed at all, which keeps the check fast on LF repositories.
Encoding. A new side made of ASCII only cannot carry encoding damage and skips the next checks. Otherwise:
- U+FFFD is the byte sequence EF BF BD; the script compares the number of lines that contain it on both sides.
- Mojibake is counted with byte patterns that real text does not contain: a UTF-8 lead byte read as cp1252 or latin-1 (Ã followed by © for é), a Cyrillic lead byte read as cp1251 (Р or С followed by a character that Russian text never puts there), and the same for dashes and quotes. Pairs that genuine Russian text does have, such as С° or С…, are left out of the pattern. Ten new sequences are needed: an article that quotes one mojibake word, 2 to 6 sequences, passes, and one damaged line of Russian has twenty or more.
- UTF-8 validity is iconv -f UTF-8 -t UTF-8: it fails on the first invalid byte and says where. Only a base that was valid counts, so old files in another encoding can still be edited.
- A BOM is the three bytes at the start of the file.
Whitespace. A file whose diff disappears under git diff -b gets the whitespace note. -b and not -w, because -w also hides a space removed between two words: the story of that bug is in A pre-commit check that blocked a correct commit.
The script carries two exceptions for the repository it was written for: a rolling feed file that a post-commit hook rewrites whole, and generated CSS bundles that inherit the line endings of their sources. Both are a case pattern on the path; change them or remove them for your repository.
Tests
The tests follow the pattern of the other gates: a scratch git repository per test, the real script run in it with --staged, and the exit status and the report checked. Damage cases are one table of base content, new content and the expected message, shortened here to the main cases. The damaged content is built at run time, so the test file itself contains no mojibake and no U+FFFD for the hook to find:
@pytest.mark.parametrize( ("base", "new", "message"), [ (crlf(LONG), EDITED, "CRLF line endings lost (base CRLF on all 150 lines, now LF) together with other changes"), (crlf(LONG), crlf(LONG) + b"added with LF\n", "mixed line endings, 1 of 151 lines now end in LF"), (LONG, crlf(LONG), "line endings flipped from LF to CRLF"), (LONG, LONG + b"pasted\r\n", "mixed line endings, 1 of 151 lines now end in CRLF"), (ORIGINAL, ORIGINAL + "bad \N{REPLACEMENT CHARACTER} byte\n".encode(), "U+FFFD replacement chars introduced"), ( RUSSIAN.encode(), RUSSIAN.encode().decode("cp1251").encode(), "mojibake introduced", ), (RUSSIAN.encode(), RUSSIAN.encode("cp1251"), "no longer valid UTF-8"), (ORIGINAL, b"\xef\xbb\xbf" + ORIGINAL, "UTF-8 BOM added"), ], ) def test_damage_is_blocked(repo, base, new, message): result = gate(repo, base, new, "damaged.txt") assert result.returncode == 1, out(result) assert message in out(result)
The same file checks that ordinary changes pass: a conversion to LF with a note, also of a mixed file and of a moved file, a 1:1 rewrite of 150 lines in LF and in CRLF, a CRLF file edited with its endings kept, a translation of 120 lines, one quoted mojibake word, an edit of an old cp1251 file. A conversion with one extra space still blocks. Other tests cover renames, a path with Cyrillic letters, the four modes and a wrong option:
python -m pytest tests/linters/test_check_diff_artifacts_sh.py -v
tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[crlf-lost-with-edit] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[crlf-lost-with-space-edit] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[crlf-file-gets-lf-line] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[lf-to-crlf] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[lf-file-gets-crlf-line] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[fffd] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[mojibake-cp1251] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[mojibake-latin1] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[saved-as-cp1251] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_damage_is_blocked[bom-added] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_a_note[trailing] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_a_note[tab-indent] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_a_note[space-run] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_space_removed_inside_a_line_passes PASSED tests/linters/test_check_diff_artifacts_sh.py::test_ordinary_changes_pass[1to1-rewrite-lf] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_ordinary_changes_pass[1to1-rewrite-crlf] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_ordinary_changes_pass[crlf-kept] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_ordinary_changes_pass[translation] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_ordinary_changes_pass[one-mojibake-sample] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_ordinary_changes_pass[legacy-cp1251-edit] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_bom_removed_is_a_note PASSED tests/linters/test_check_diff_artifacts_sh.py::test_new_file_findings_are_notes[mixed-endings] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_new_file_findings_are_notes[fffd] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_new_file_findings_are_notes[bom] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_new_php_file_with_bom_is_blocked PASSED tests/linters/test_check_diff_artifacts_sh.py::test_conversion_to_lf_is_a_note[crlf] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_conversion_to_lf_is_a_note[mostly-crlf-mixed] PASSED tests/linters/test_check_diff_artifacts_sh.py::test_renamed_and_converted_file_is_a_note PASSED tests/linters/test_check_diff_artifacts_sh.py::test_feed_json_line_endings_are_exempt PASSED tests/linters/test_check_diff_artifacts_sh.py::test_generated_css_line_endings_are_a_note PASSED tests/linters/test_check_diff_artifacts_sh.py::test_binary_file_is_skipped PASSED tests/linters/test_check_diff_artifacts_sh.py::test_renamed_file_is_compared_with_its_source PASSED tests/linters/test_check_diff_artifacts_sh.py::test_non_ascii_path_is_checked PASSED tests/linters/test_check_diff_artifacts_sh.py::test_all_mode_sees_unstaged_changes PASSED tests/linters/test_check_diff_artifacts_sh.py::test_commit_mode_audits_a_commit PASSED tests/linters/test_check_diff_artifacts_sh.py::test_unknown_argument_is_a_usage_error PASSED ============================= 36 passed in 22.90s ==============================
Limits
- Small mojibake passes. Fewer than ten new sequences, such as one damaged short title, is under the threshold. A page that quotes a whole sentence of mojibake on purpose is over it and needs the skip.
- A move with a big rewrite is a new file. Below 50% similarity git does not pair the two paths, and the new file only gets the new-file checks.
- New files are not judged. Mixed endings, U+FFFD or a BOM in a new file are notes, except a BOM in a new PHP file.
- Mixed files stay mixed. A file that was already mixed only fails when it was mostly CRLF and becomes all LF.
- A conversion and an edit of one file go in two commits. In one commit the conversion hides the edit, and the hook blocks it.
- A committed damage is repaired by a flip. Undo an unpushed commit with git reset --soft HEAD~1; after a push, commit the repair with the skip.
- Requirements. GNU grep with -P for the mojibake rule and iconv for the UTF-8 rule. Without one of them that rule is skipped with a message on standard error.
The script
The whole script, version 0.2.0. Save it as tools/check-diff-artifacts.sh and make it executable, as in Setting it up:
#!/usr/bin/env bash # version 0.2.0 # Block the byte-level damage that scripted rewrites (sed, Python # read/write round trips, editors) leave in a diff, and nothing else. # # Blocking (ARTIFACT, exit 1), each judged against the base side: # 1. line endings: CRLF lost in a CRLF file together with other changes, # an LF file flipped to CRLF, or a file with one ending style left with # mixed endings. A file whose only change is CRLF -> LF passes with a # NOTE, so a repository can move to LF in conversion-only commits. # 2. U+FFFD replacement chars introduced into an existing file (a lossy # decode) # 3. mojibake introduced: UTF-8 decoded as cp1252/latin-1 or cp1251 and # written back, 10 or more new sequences (an e-acute turned into # A-tilde + copyright sign, Russian turned into runs of R/S pairs) # 4. a valid UTF-8 file no longer valid UTF-8 (saved as cp1251/latin-1) # 5. a UTF-8 BOM added to an existing file or to a new PHP file (PHP # prints it before any header) # Not blocking (NOTE, exit stays 0): whitespace-only diffs, a BOM removed, # stray CRs dropped from a mostly-LF file, and a new file with mixed endings, # U+FFFD or (not PHP) a BOM: a new file has no base to be damaged against. # A 1:1 rewrite of many lines (translation, path rename, regenerated # sitemap) is an ordinary change and is not reported: the EOL and encoding # checks above catch the damage the old "full-file rewrite" rule stood for. # The rolling feed JSON files skip the line-ending and whitespace checks: # the feed writer serializes them whole. Generated CSS concatenations # (css_pipeline/build/) take their line endings from their sources, which # are checked themselves, so a line-ending hit there is only a NOTE. # # Usage: # check-diff-artifacts.sh [--staged | --all | --commit REV] # (default) unstaged changes: index -> working tree # --staged what the next commit records: HEAD -> index (pre-commit) # --all HEAD -> working tree, staged and unstaged together # --commit REV an existing commit: its first parent -> REV # Exit 0 when clean (notes allowed), 1 when artifacts are found, 2 on a # usage error. Untracked, deleted, binary, symlink and submodule entries # are skipped. A renamed or copied file is compared with its source. set -u usage() { sed -n '/^# Usage:/,/^# are skipped/{s/^# \{0,1\}//;p}' "$0"; } MODE="worktree" REV="" while [ $# -gt 0 ]; do case "$1" in --staged) MODE="staged" ;; --all) MODE="all" ;; --commit) if [ $# -lt 2 ]; then usage >&2; exit 2; fi MODE="commit"; REV="$2"; shift ;; --commit=*) MODE="commit"; REV="${1#--commit=}" ;; -h|--help) usage; exit 0 ;; *) echo "check-diff-artifacts: unknown argument: $1" >&2; usage >&2; exit 2 ;; esac shift done cd "$(git rev-parse --show-toplevel)" || exit 2 EMPTY_TREE=$(git hash-object -t tree /dev/null) head_or_empty() { if git rev-parse -q --verify HEAD >/dev/null; then echo HEAD; else echo "$EMPTY_TREE"; fi } # The same pairs for --raw (blob ids), --numstat and -b --numstat. case "$MODE" in worktree) LABEL="worktree"; RANGE=(--diff-filter=ACM) ;; staged) LABEL="staged"; RANGE=(--cached -M --diff-filter=ACMR "$(head_or_empty)") ;; all) LABEL="all"; RANGE=(-M --diff-filter=ACMR "$(head_or_empty)") ;; commit) if ! COMMIT=$(git rev-parse -q --verify "$REV^{commit}"); then echo "check-diff-artifacts: not a commit: $REV" >&2; exit 2 fi PARENT=$(git rev-parse -q --verify "$COMMIT^1") || PARENT="$EMPTY_TREE" LABEL="commit ${COMMIT:0:9}" RANGE=(-M --diff-filter=ACMR "$PARENT" "$COMMIT") ;; esac GIT_DIFF=(git diff -z --no-ext-diff --no-color) TMP=$(mktemp -d) || exit 2 trap 'rm -rf "$TMP"' EXIT HAVE_PCRE=1 printf 'x\n' | LC_ALL=C grep -qP 'x' 2>/dev/null || HAVE_PCRE=0 HAVE_ICONV=1 command -v iconv >/dev/null 2>&1 || HAVE_ICONV=0 # Double-encoded UTF-8, as bytes (this file stays ASCII so it never matches # itself). Latin: a lead byte C2-DF read as cp1252 or latin-1 (U+00C2-U+00DF) # followed by what its continuation byte turns into; E2 80 xx punctuation # (dashes, quotes) read as cp1252 or latin-1. Cyrillic: a lead D0/D1 read as # cp1251 (U+0420/U+0421) followed by what the second byte turns into, minus # the pairs real Russian text has (before NBSP, guillemets, degree, ellipsis, # dashes, quotes, yo); E2 80 xx punctuation read as cp1251. MOJIBAKE='\xC3[\x82-\x9F](?:\xC2[\x80-\xBF]|\xE2\x80[\x93-\xBA]|\xE2\x82\xAC|\xE2\x84\xA2|\xC5[\x92\x93\xA0\xA1\xB8\xBD\xBE]|\xC6\x92|\xCB[\x86\x9C])|\xC3\xA2(?:\xE2\x82\xAC|\xC2[\x80-\x9F])|\xD0[\xA0\xA1](?:\xD0[\x82-\x8F]|\xD1[\x92-\x9F]|\xD2[\x90\x91]|\xC2[\xA4-\xAA\xAC-\xAF\xB1-\xBA\xBC-\xBF]|\xE2\x80[\x9A\x9E\xA0\xA1\xA2\xB0\xB9\xBA]|\xE2\x82\xAC|\xE2\x84[\x96\xA2])|\xD0\xB2\xD0\x82' # New sequences needed: an article may quote a short sample (one mojibake # word is 2-6 sequences); one damaged line of Russian is 20 or more. MOJIBAKE_MIN=10 FOUND=0 NOTES=0 report() { echo "ARTIFACT: $1"; FOUND=$((FOUND + 1)); } note() { echo "NOTE: $1"; NOTES=$((NOTES + 1)); } is_zero() { case "$1" in *[!0]*) return 1 ;; *) return 0 ;; esac; } lines() { local n; n=$(wc -l < "$1"); echo $((n)); } cr_lines() { LC_ALL=C grep -ac $'\r$' "$1" || true; } non_ascii() { LC_ALL=C grep -aq $'[\x80-\xff]' "$1"; } fffd() { LC_ALL=C grep -ac $'\xef\xbf\xbd' "$1" || true; } mojibake() { local n; n=$(LC_ALL=C grep -aoP "$MOJIBAKE" "$1" 2>/dev/null | wc -l); echo $((n)); } has_bom() { local head="" IFS= read -r -n 3 head < "$1" || true # 3 bytes, or 1-3 chars in a UTF-8 locale [[ $head == $'\xef\xbb\xbf'* ]] } utf8_error() { # prints iconv's complaint and fails when $1 is not UTF-8 local err err=$(iconv -f UTF-8 -t UTF-8 "$1" 2>&1 >/dev/null) && return 0 printf '%s' "${err##*: }" return 1 } # True when $2 is exactly $1 with the CR before each line break removed: # a conversion to LF and nothing else. only_cr_removed() { LC_ALL=C sed 's/\r$//' "$1" | cmp -s - "$2"; } # Ending style of a file: none (no line break), lf, crlf or mixed. eol_style() { local n="$1" c="$2" if [ "$n" -eq 0 ]; then echo none elif [ "$c" -eq 0 ]; then echo lf elif [ "$c" -ge "$n" ]; then echo crlf else echo mixed fi } # $1 path, $2 display name, $3 base file or "", $4 new file. check_eol() { local path="$1" f="$2" base="$3" new="$4" bn=0 bc=0 bs="absent" nn nc ns hit=report case "$path" in container/css_pipeline/build/*) hit=note ;; esac nc=$(cr_lines "$new") [ -z "$base" ] || bc=$(cr_lines "$base") [ "$nc" -gt 0 ] || [ "$bc" -gt 0 ] || return 0 # LF on both sides if [ -n "$base" ]; then bn=$(lines "$base"); bs=$(eol_style "$bn" "$bc") fi nn=$(lines "$new"); ns=$(eol_style "$nn" "$nc") case "$bs:$ns" in crlf:lf) if only_cr_removed "$base" "$new"; then note "$f [$LABEL]: converted from CRLF to LF, nothing else changed ($nn lines)" else "$hit" "$f [$LABEL]: CRLF line endings lost (base CRLF on all $bn lines, now LF) together with other changes; a conversion alone passes" fi ;; crlf:mixed) "$hit" "$f [$LABEL]: mixed line endings, $((nn - nc)) of $nn lines now end in LF in a CRLF file" ;; lf:crlf) "$hit" "$f [$LABEL]: line endings flipped from LF to CRLF ($nn lines)" ;; lf:mixed) "$hit" "$f [$LABEL]: mixed line endings, $nc of $nn lines now end in CRLF in an LF file" ;; mixed:lf) if only_cr_removed "$base" "$new"; then note "$f [$LABEL]: converted from mixed endings to LF, nothing else changed ($bc of $bn lines had CRLF)" elif [ $((bc * 2)) -ge "$bn" ]; then "$hit" "$f [$LABEL]: CRLF line endings lost ($bc of $bn base lines had CRLF, now none) together with other changes; a conversion alone passes" else note "$f [$LABEL]: stray CRs removed ($bc of $bn base lines), the file is LF now" fi ;; absent:mixed) note "$f [$LABEL]: new file with mixed line endings ($nc of $nn lines CRLF)" ;; esac } # $1 path, $2 display name, $3 base file or "", $4 new file. check_encoding() { local path="$1" f="$2" base="$3" new="$4" b n err if ! non_ascii "$new"; then # plain ASCII: valid UTF-8, nothing to damage if [ -n "$base" ] && has_bom "$base"; then note "$f [$LABEL]: UTF-8 BOM removed"; fi return 0 fi n=$(fffd "$new") if [ "$n" -gt 0 ] && [ -z "$base" ]; then # A new page may show the character on purpose (unichr, /dev/urandom). note "$f [$LABEL]: new file with U+FFFD on $n line(s)" elif [ "$n" -gt 0 ]; then b=$(fffd "$base") if [ "$n" -gt "$b" ]; then report "$f [$LABEL]: U+FFFD replacement chars introduced on $((n - b)) more line(s) ($b -> $n, a lossy decode?)" fi fi if [ "$HAVE_PCRE" -eq 1 ]; then n=$(mojibake "$new") if [ "$n" -ge "$MOJIBAKE_MIN" ]; then b=0; [ -z "$base" ] || b=$(mojibake "$base") if [ $((n - b)) -ge "$MOJIBAKE_MIN" ]; then report "$f [$LABEL]: mojibake introduced ($b -> $n double-encoded sequences: UTF-8 read as cp1252, latin-1 or cp1251 and saved again)" fi fi fi if [ "$HAVE_ICONV" -eq 1 ] && [ -n "$base" ] && ! err=$(utf8_error "$new"); then if utf8_error "$base" >/dev/null; then report "$f [$LABEL]: no longer valid UTF-8 (${err:-invalid byte}), saved in another encoding?" fi fi if has_bom "$new"; then if [ -n "$base" ] && ! has_bom "$base"; then report "$f [$LABEL]: UTF-8 BOM added (PHP prints it before any header)" elif [ -z "$base" ]; then case "$path" in *.php) report "$f [$LABEL]: new PHP file with a UTF-8 BOM (printed before any header)" ;; *) note "$f [$LABEL]: new file with a UTF-8 BOM" ;; esac fi elif [ -n "$base" ] && has_bom "$base"; then note "$f [$LABEL]: UTF-8 BOM removed" fi } # $1 path, $2 source path (rename/copy) or "", $3 base sha (zeros: none), # $4 new sha (zeros: read the working tree file). check_file() { local f="$1" src="$2" ssha="$3" dsha="$4" name base="" new stat adds dels before notes_before eol_notes name="$f" [ -z "$src" ] || [ "$src" = "$f" ] || name="$f (from $src)" if ! is_zero "$ssha"; then base="$TMP/base" git cat-file blob "$ssha" > "$base" 2>/dev/null || return 0 fi if is_zero "$dsha"; then [ -f "$f" ] && [ ! -L "$f" ] || return 0 new="$f" else new="$TMP/new" git cat-file blob "$dsha" > "$new" 2>/dev/null || return 0 fi # git's own text/binary call (NUL in the first 8000 bytes, attributes): # binary files show "-" counts and none of the line or text checks apply. stat="${NUMSTAT[$f]:-}" [ -n "$stat" ] || return 0 read -r adds dels <<< "$stat" [ "$adds" != "-" ] && [ "$dels" != "-" ] || return 0 before=$FOUND notes_before=$NOTES case "$f" in container/src/feed/feed.json|container/src/feed/arch.json) ;; *) check_eol "$f" "$name" "$base" "$new" ;; esac eol_notes=$((NOTES - notes_before)) check_encoding "$f" "$name" "$base" "$new" # A line-ending note already explains a whitespace-only diff. if [ "$FOUND" -eq "$before" ] && [ "$eol_notes" -eq 0 ] && [ -n "$base" ]; then case "$f" in container/src/feed/feed.json|container/src/feed/arch.json) ;; *) # -b, not -w: -w also hides a space added or removed inside # a line ("Java Script" -> "JavaScript"), a real text change. if [ -z "${NONWS[$f]:-}" ]; then note "$name [$LABEL]: whitespace-only change ($adds+/${dels}-: indentation, trailing spaces), not blocking" fi ;; esac fi } # numstat -z: "adds<TAB>dels<TAB>path" or, for a rename or copy, # "adds<TAB>dels<TAB>" then the source and the destination as two fields. declare -A NUMSTAT=() NONWS=() read_numstat() { # $1: NUMSTAT or NONWS, the array to fill; the rest: diff options local which="$1" rec adds dels path shift while IFS= read -r -d '' rec; do IFS=$'\t' read -r adds dels path <<< "$rec" if [ -z "$path" ]; then IFS= read -r -d '' _ || break # rename/copy source IFS= read -r -d '' path || break fi if [ "$which" = NONWS ]; then NONWS["$path"]="$adds $dels" else NUMSTAT["$path"]="$adds $dels" fi done < <("${GIT_DIFF[@]}" --numstat "$@" "${RANGE[@]}") } read_numstat NUMSTAT read_numstat NONWS -b # -b keeps a whitespace-only change listed with 0 0: drop those. for k in "${!NONWS[@]}"; do [ "${NONWS[$k]}" != "0 0" ] || unset 'NONWS[$k]' done while IFS= read -r -d '' meta; do IFS= read -r -d '' path || break src="" read -r smode dmode ssha dsha status <<< "${meta#:}" case "$status" in R*|C*) src="$path"; IFS= read -r -d '' path || break ;; esac case "$smode:$dmode" in 120000:*|*:120000|160000:*|*:160000) continue ;; esac [ "$ssha" != "$dsha" ] || continue # pure rename or mode change check_file "$path" "$src" "$ssha" "$dsha" done < <("${GIT_DIFF[@]}" --raw --no-abbrev "${RANGE[@]}") if [ "$HAVE_PCRE" -eq 0 ]; then echo "check-diff-artifacts: grep -P is not available, the mojibake check was skipped" >&2 fi if [ "$HAVE_ICONV" -eq 0 ]; then echo "check-diff-artifacts: iconv is not available, the UTF-8 validity check was skipped" >&2 fi if [ "$FOUND" -gt 0 ]; then echo "check-diff-artifacts: $FOUND artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md" exit 1 fi if [ "$NOTES" -gt 0 ]; then echo "check-diff-artifacts: clean ($NOTES note(s), not blocking)" else echo "check-diff-artifacts: clean" fi exit 0
More on hooks that look only at what is staged is in linting what is staged, and on the order of hooks that write files in pre-commit and post-commit together.
Article author: Klavdiya Kondakova