Reworking a pre-commit check so it fires only on real damage
A pre-commit check is worth as much as its record: it should fail on damage and on nothing else. This page is about one check that did neither well, and how it was rebuilt and tested. The check, check-diff-artifacts.sh, guards a PHP site with English and Russian pages, about 11,000 files in one git repository. It reads the staged diff for the damage that scripted edits leave behind. How the current version is used is in A pre-commit check for line-ending and encoding damage; the first version and its git diff -w bug are in A pre-commit check that blocked a correct commit.
What we had
Version 0.0.4 had four rules, each checked on every staged file:
- a whitespace or line-ending only diff: the diff is empty under git diff -b;
- CRLF lost: the HEAD version has a CR and the new one has none;
- U+FFFD replacement characters added;
- a full-file rewrite: as many lines added as deleted, in a file of more than 100 lines.
Rules 2 and 3 describe damage. Rules 1 and 4 describe what damage often looks like: a CRLF flip changes every line and nothing but whitespace, so a diff with equal counts or a diff that vanishes under -b might be one. When a rule fired on a change that was fine, the working notes of the repository prescribed an audit: compare the CR counts of every staged file with HEAD, grep for U+FFFD, check that git diff -w is empty, then commit with SKIP=check-diff-artifacts and a sentence about the audit in the commit body.
Problem 1: blocks on ordinary work
A site in two languages does a lot of 1:1 rewrites. A page translated line by line has as many added lines as deleted ones, and so has a sitemap whose dates change, or a page where an image folder is renamed in every link. Rule 4 fired on all of them. Here the first version meets a translated page of 150 lines in a scratch repository. No hook is installed there: the old script, saved as tools/old-check.sh, and the new one are run by hand. The two versions lie next to the repository as old.sh and new.sh.
git init -q demo && cd demo && git config user.email you@example.com && git config user.name You
mkdir -p tools && cp ../old.sh tools/old-check.sh && cp ../new.sh tools/check-diff-artifacts.sh
for i in $(seq 1 150); do echo "<p>Строка $i: текст страницы</p>"; done > guide.html
printf 'function sum(a, b) {\r\n return a + b;\r\n}\r\n' > sum.js
git add . && git commit -qm "Adds the pages"
for i in $(seq 1 150); do echo "<p>Line $i: the text of the page</p>"; done > guide.html
git add guide.html
git diff --cached --numstat
150 150 guide.html
tools/old-check.sh --staged
ARTIFACT: guide.html [staged]: full-file rewrite (150 lines removed+added, possible EOL flip)
tools/check-diff-artifacts.sh --staged
check-diff-artifacts: clean
git commit -qm "Translates the guide"
Replayed over the history of the repository (see below), the first version blocked 16 commits. Two of them were real damage. The other 14 were ordinary work:
| Kind of change | Commits | Rule |
|---|---|---|
| a long page translated line by line | 4 | 4 |
| sitemap regenerated with new dates, up to 5,228 lines | 3 | 4 |
| a path or URL changed on every line of a page | 3 | 4 |
| code blocks reformatted or recoloured | 2 | 4 |
| an intended whitespace edit | 2 | 1 and 4 |
The same rules also made one piece of ordinary work impossible: moving the repository to LF. A commit that only converts a CRLF file to LF changes every line and nothing but line endings, so rules 1 and 2 both fire, and the repository was stuck with 309 CRLF files next to more than 10,000 LF ones. The fix section shows such a commit with both versions.
Eight commit messages in the history end with the audit sentence. A check that is skipped every few days teaches the team to skip it, and the audit, seven commands by hand, was done less carefully each time.
Problem 2: damage that passed
The same replay found damage the first version let through, and a closer look at the script found the reasons:
- Renamed files were skipped. The file list came from git diff --name-only --diff-filter=ACM; a rename has the status R. A move of six example files that also turned their CRLF into LF went in unnoticed.
- Paths with Cyrillic letters were skipped. Git quotes such a name in its output, and git show HEAD:"\320\263..." finds no file. Ten files of the repository have such names.
- Only a total loss of CRLF counted. Rule 2 needed every CR gone; a line appended with LF to a CRLF file passed, and so did an LF file flipped to CRLF as part of a real edit.
- Encoding damage other than U+FFFD had no rule. Mojibake (garbled text: bytes of one encoding read as another and saved again) or a file saved as cp1251 showed up only through rule 4, when the file was long enough and the counts happened to match. A BOM changes one line and had no rule.
- --all did not see unstaged changes. It compared HEAD with the index for every file, the same as --staged.
Each case reproduced in the scratch repository: a move that loses CRLF and adds a line, a file with a Cyrillic name that loses CRLF and gets a line, an unstaged loss with an edit under --all. The old script reports all three as clean:
mkdir -p examples && git mv sum.js examples/sum.js
sed -i 's/\r$//; 1i // Adds two numbers' examples/sum.js && git add examples/sum.js
git status --short
R sum.js -> examples/sum.js
tools/old-check.sh --staged
check-diff-artifacts: clean
tools/check-diff-artifacts.sh --staged
ARTIFACT: examples/sum.js (from sum.js) [staged]: CRLF line endings lost (base CRLF on all 3 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
The move is committed with its CRLF back, and a list of cities with a Cyrillic file name follows:
sed -i 's/$/\r/' examples/sum.js && git add examples/sum.js && git commit -qm "Moves sum.js to examples/"
printf 'Москва\r\nТампере\r\n' > города.txt && git add города.txt && git commit -qm "Adds the city list"
printf 'Москва\nТампере\nТурку\n' > города.txt && git add города.txt
git diff --cached --name-only
"\320\263\320\276\321\200\320\276\320\264\320\260.txt"
tools/old-check.sh --staged
check-diff-artifacts: clean
tools/check-diff-artifacts.sh --staged
ARTIFACT: города.txt [staged]: CRLF line endings lost (base CRLF on all 2 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
git commit -qm "Adds Turku"
Last, a CRLF loss with an edit that is not staged:
sed -i 's/\r$//; s/two numbers/two numbers a and b/' examples/sum.js
tools/old-check.sh --all
check-diff-artifacts: clean
tools/check-diff-artifacts.sh --all
ARTIFACT: examples/sum.js [all]: CRLF line endings lost (base CRLF on all 4 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
The fix: block only what is never intended
The two problems have one cause: rules 1 and 4 measure the shape of a diff, and both damage and ordinary work can have that shape. The new version measures the damage itself. Each rule compares the bytes of the new side with the bytes of the base side, and blocks only what nobody ever does on purpose in this repository. What deserves attention but is not damage becomes a NOTE that does not block.
The line-ending rule shows the idea best. Losing CRLF is damage when it hides an edit, because a diff where every line changed cannot be reviewed. A conversion with nothing else hides nothing, and that can be proved: the new file must be exactly the base with the CR before each line break removed. The new version checks this with one comparison and lets such a file through with a note, so the repository can move to LF in commits that only convert. On the scratch repository, the old version blocks a pure conversion twice and the new one passes it:
git checkout -- examples/sum.js
sed -i 's/\r$//' examples/sum.js && git add examples/sum.js
tools/old-check.sh --staged
ARTIFACT: examples/sum.js [staged]: whitespace/EOL-only diff (4+/ 4-, invisible under -b) ARTIFACT: examples/sum.js [staged]: CRLF line endings lost (HEAD blob has CR)
tools/check-diff-artifacts.sh --staged
NOTE: examples/sum.js [staged]: converted from CRLF to LF, nothing else changed (4 lines) check-diff-artifacts: clean (1 note(s), not blocking)
A conversion mixed with an edit, as in all three cases above, still blocks, and the message says what would pass. The exception is one way: LF to CRLF always blocks, because the goal is a repository that ends up LF.
| 0.0.4 | 0.2.0 |
|---|---|
| whitespace or line-ending only diff: blocks | a line-ending change is judged by the line-ending rule; any other whitespace-only diff is a NOTE |
| CRLF lost: every CR gone | the ending style of each side (lf, crlf, mixed): CRLF lost together with other changes, mixed endings in a file that had one style, LF flipped to CRLF; a pure conversion to LF is a NOTE |
| U+FFFD added | kept for existing files; a new file with U+FFFD is a NOTE |
| full-file rewrite: blocks | removed; mojibake, UTF-8 validity and BOM rules check for the damage it stood for |
| renames and Cyrillic paths skipped | git diff --raw -z -M: renames compared with their source, paths unquoted |
| --all = --staged | --all compares HEAD with the working tree; new --commit REV |
How each rule works is in How the checks work. The --commit mode was added first, because it is what makes the next test possible.
Testing on the whole history
Unit tests check the cases someone thought of. The history of the repository holds the cases that really happened: every commit is a staged diff that the check saw, or would have seen. The test is to run both versions on every commit and read every finding. The new version does it with --commit; a small wrapper prefixes each finding with the commit id:
vi replay.sh
#!/usr/bin/env bash # replay.sh REV: run the check on one commit, prefix each finding with REV out=$(container/tools/check-diff-artifacts.sh --commit "$1" 2>&1) grep -E '^(ARTIFACT|NOTE):' <<< "$out" | sed "s/^/$1 /"
chmod +x replay.sh
git rev-list --no-merges HEAD > commits.txt
wc -l < commits.txt
1174
xargs -P 8 runs eight commits at a time; one commit per call (-n 1):
time (xargs -P 8 -n 1 ./replay.sh < commits.txt > hits.txt)
real 16m15,641s user 11m58,344s sys 21m46,361s
Version 0.0.4 had no way to check a commit, so for the old rules a second wrapper ran the same four tests on the pair REV^ and REV with git diff REV^ REV and git show REV:path.
Results of the replay
The commits that the new version blocks:
awk '$2 == "ARTIFACT:" {print substr($1, 1, 9)}' hits.txt | sort | uniq -c | sort -rn
275 2af4d0554 6 45e2c4d58 4 8d889c516 1 b68ebc079 1 a402d2992
Every one of them was read:
- 275 files: a commit that fixed the code style of the site with a formatter, which turned every CRLF file it touched into LF while it edited them. Real damage; the old version caught it too. It counted 332, because it also blocked 53 files of the same commit whose only change was the line endings, and 4 mixed text dumps normalised to LF: those 57 are now notes.
- 6 files: the move of the six example files. Real; the old version missed it.
- 4 files and 1 file: generated CSS that became mixed because the build put a CRLF source file into an LF bundle. These were the old build folders; the same case under today's build folder is a note, because the bundle takes its endings from its sources and the sources are checked themselves.
- 1 file: a refactor that lost CRLF in one include file. Real; the old version caught it.
None of the 14 false positives is among them. The notes were read too:
awk '$2 == "NOTE:"' hits.txt | sed -E 's/^[0-9a-f]+ NOTE: [^[]*\[[^]]*\]: //; s/(^|[ (/])[0-9]+/\1N/g' | sort | uniq -c | sort -rn
53 converted from CRLF to LF, nothing else changed (N lines) 16 whitespace-only change (N+/N-: indentation, trailing spaces), not blocking 8 new file with mixed line endings (N of N lines CRLF) 7 new file with U+FFFD on N line(s) 4 converted from mixed endings to LF, nothing else changed (N of N lines had CRLF) 2 new file with a UTF-8 BOM
Two of these kinds were blocks in the first draft of the new version and came out of the replay as wrong. The first commit of the repository, an import of 9,155 files, has pages that show U+FFFD on purpose (a page about chr(), output of /dev/urandom), and a vendored library ships READMEs with a BOM. A new file has no base to be damaged against, so for new files these are now notes, except a BOM in a new PHP file, which breaks the headers of the page.
Calibrating the mojibake rule
Mojibake is UTF-8 read in a one-byte encoding and saved again. The rule counts byte patterns of such text: a UTF-8 lead byte read as cp1252 or latin-1, and the Cyrillic lead bytes read as cp1251, which turn every Russian letter into Р or С followed by a character that Russian does not put there. The first version of the pattern was run on every tracked text file of the repository, where no mojibake was expected. It found genuine text: a mathematics dictionary writes "category С°" and "С…" before an ellipsis. Those pairs went out of the pattern. With the final pattern of the script in the variable MOJIBAKE, the search finds three files:
LC_ALL=C git grep -c -I -P "$MOJIBAKE"
container/src/dev/python/collections/string/index.php:1 container/src/dev/python/strings.php:1 container/src/ru/code/python/objects/str/index.php:1
All three are Python lessons that quote one mojibake word to show what a wrong decode looks like. Such a quote is 2 to 6 sequences, and one damaged line of Russian is 20 or more, so the rule needs ten new sequences in a file. The quotes pass, a damaged page does not.
The check on its own change
The first run of the new version over the working tree found three artifacts, all in the change itself:
ARTIFACT: container/tests/linters/test_check_diff_artifacts_sh.py [all]: U+FFFD replacement chars introduced (0 -> 1, a lossy decode?) ARTIFACT: container/tools/CHECK_DIFF_ARTIFACTS.md [all]: mojibake introduced (0 -> 6 sequences like é or Рё: UTF-8 read as cp1252, latin-1 or cp1251 and saved again) ARTIFACT: container/tools/check-diff-artifacts.sh [all]: mojibake introduced (0 -> 7 sequences like é or Рё: UTF-8 read as cp1252, latin-1 or cp1251 and saved again) check-diff-artifacts: 3 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md
The test file had a literal U+FFFD in a test case, and the documentation and the script quoted mojibake as examples. A check that cannot commit itself would have been skipped on its first day. The fix: the script is ASCII only, the documentation describes the examples instead of quoting them, the tests build the damaged bytes at run time ("\N{REPLACEMENT CHARACTER}", .encode().decode("cp1251").encode()), and the threshold of ten covers a short quote. This page shows its mojibake samples as numeric HTML entities, so its source passes the check too.
Tests in scratch repositories
The self-test of the script grew from 5 cases to 36. Each one creates a git repository in a temporary folder, commits a base file, stages a new version and runs the real script:
- 10 damage cases that must block: CRLF lost with an edited line and with one extra space, an LF line in a CRLF file, LF to CRLF, a CRLF line in an LF file, U+FFFD, cp1251 and latin-1 mojibake, a file saved as cp1251, a BOM added;
- 13 ordinary changes that must pass: conversions to LF of a CRLF file, of a mostly-CRLF mixed file and of a moved file (with a note), whitespace-only edits (with a note), a space removed inside a line, 1:1 rewrites in LF and CRLF, a translation, a quoted mojibake word, an edit of an old cp1251 file;
- 13 cases for the edges: notes for a BOM removed and for new files, a BOM in a new PHP file, the two exempt paths, a binary file, a rename, a Cyrillic path, --all and the default mode, --commit, a wrong option.
The list with the output of the run is in Tests.
Speed
A pre-commit check runs on every commit, so its time is paid every time. The first draft of the new version started about twenty processes per file: two git diff --no-index calls, line counts, greps and iconv. Three changes cut that down: one git diff --numstat and one git diff -b --numstat for all files instead of two calls per file, no line counting when neither side has a CR, and no encoding checks when the new side is plain ASCII. Measured with --commit on the same machine under the same load:
| Commit | First draft | Final |
|---|---|---|
| 48 files | 8.7 s | 1.7 s |
| 17 files | 3.7 s | 2.0 s |
| 382 files | 63.5 s | 21.0 s |
A linter that checked nothing
Before the commit, all hooks were run on the changed files with pre-commit run --files, and ShellCheck passed. The commit itself failed:
shellcheck (maintained .sh)..............................................Failed - hook id: shellcheck - exit code: 1 linting 1 shell script(s) with shellcheck ... In container/tools/check-diff-artifacts.sh line 258: into["$path"]="$adds $dels" ^-----------^ SC2034 (warning): into appears unused. Verify use (or export if used externally).
The shellcheck hook of the repository is configured with pass_filenames: false and lints the staged scripts itself. Nothing was staged during --files, so it linted nothing and passed. The warning is a known false alarm for a bash nameref (local -n into=...), and the function was rewritten without one. The real check was the hook's own linter on the file, together with the same linter on the old staged copy, which had to fail with SC2034 to prove that the file was really read.
What we learned
- Check for the damage, not for its shape. A rule that stands for damage ("equal counts might be an EOL flip") fires on everything with the same shape. Rule 4 had 14 false positives and caught nothing that rule 2 did not.
- Replay the history before trusting a gate. One mode that runs the check on an existing commit turns the whole history into a test set. Every false positive and every miss on this page came from that replay.
- Let the safe form of a blocked change through. A CRLF loss is blocked because it can hide an edit; a conversion that provably hides none passes, and with it the repository can move to LF.
- Notes for what is not damage. A whitespace-only edit deserves a look, not a block.
- A check must pass its own change. Documentation and tests of a check quote the very thing it looks for.
- Make sure a linter read the file. A passing hook that lints nothing looks the same as a passing hook that lints everything.
Article author: Klavdiya Kondakova