Reworking a pre-commit check so it fires only on real damage

A pre-commit check is worth as much as its record: it should fail on damage and on nothing else. This page is about one check that did neither well, and how it was rebuilt and tested. The check, check-diff-artifacts.sh, guards a PHP site with English and Russian pages, about 11,000 files in one git repository. It reads the staged diff for the damage that scripted edits leave behind. How the current version is used is in A pre-commit check for line-ending and encoding damage; the first version and its git diff -w bug are in A pre-commit check that blocked a correct commit.

What we had

Version 0.0.4 had four rules, each checked on every staged file:

  1. a whitespace or line-ending only diff: the diff is empty under git diff -b;
  2. CRLF lost: the HEAD version has a CR and the new one has none;
  3. U+FFFD replacement characters added;
  4. a full-file rewrite: as many lines added as deleted, in a file of more than 100 lines.

Rules 2 and 3 describe damage. Rules 1 and 4 describe what damage often looks like: a CRLF flip changes every line and nothing but whitespace, so a diff with equal counts or a diff that vanishes under -b might be one. When a rule fired on a change that was fine, the working notes of the repository prescribed an audit: compare the CR counts of every staged file with HEAD, grep for U+FFFD, check that git diff -w is empty, then commit with SKIP=check-diff-artifacts and a sentence about the audit in the commit body.

Problem 1: blocks on ordinary work

A site in two languages does a lot of 1:1 rewrites. A page translated line by line has as many added lines as deleted ones, and so has a sitemap whose dates change, or a page where an image folder is renamed in every link. Rule 4 fired on all of them. Here the first version meets a translated page of 150 lines in a scratch repository. No hook is installed there: the old script, saved as tools/old-check.sh, and the new one are run by hand. The two versions lie next to the repository as old.sh and new.sh.

git init -q demo && cd demo && git config user.email you@example.com && git config user.name You

mkdir -p tools && cp ../old.sh tools/old-check.sh && cp ../new.sh tools/check-diff-artifacts.sh

for i in $(seq 1 150); do echo "<p>Строка $i: текст страницы</p>"; done > guide.html

printf 'function sum(a, b) {\r\n return a + b;\r\n}\r\n' > sum.js

git add . && git commit -qm "Adds the pages"

for i in $(seq 1 150); do echo "<p>Line $i: the text of the page</p>"; done > guide.html

git add guide.html

git diff --cached --numstat

150 150 guide.html

tools/old-check.sh --staged

ARTIFACT: guide.html [staged]: full-file rewrite (150 lines removed+added, possible EOL flip)

tools/check-diff-artifacts.sh --staged

check-diff-artifacts: clean

git commit -qm "Translates the guide"

Replayed over the history of the repository (see below), the first version blocked 16 commits. Two of them were real damage. The other 14 were ordinary work:

The 14 false positives of version 0.0.4
Kind of changeCommitsRule
a long page translated line by line44
sitemap regenerated with new dates, up to 5,228 lines34
a path or URL changed on every line of a page34
code blocks reformatted or recoloured24
an intended whitespace edit21 and 4

The same rules also made one piece of ordinary work impossible: moving the repository to LF. A commit that only converts a CRLF file to LF changes every line and nothing but line endings, so rules 1 and 2 both fire, and the repository was stuck with 309 CRLF files next to more than 10,000 LF ones. The fix section shows such a commit with both versions.

Eight commit messages in the history end with the audit sentence. A check that is skipped every few days teaches the team to skip it, and the audit, seven commands by hand, was done less carefully each time.

Problem 2: damage that passed

The same replay found damage the first version let through, and a closer look at the script found the reasons:

Each case reproduced in the scratch repository: a move that loses CRLF and adds a line, a file with a Cyrillic name that loses CRLF and gets a line, an unstaged loss with an edit under --all. The old script reports all three as clean:

mkdir -p examples && git mv sum.js examples/sum.js

sed -i 's/\r$//; 1i // Adds two numbers' examples/sum.js && git add examples/sum.js

git status --short

R sum.js -> examples/sum.js

tools/old-check.sh --staged

check-diff-artifacts: clean

tools/check-diff-artifacts.sh --staged

ARTIFACT: examples/sum.js (from sum.js) [staged]: CRLF line endings lost (base CRLF on all 3 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md

The move is committed with its CRLF back, and a list of cities with a Cyrillic file name follows:

sed -i 's/$/\r/' examples/sum.js && git add examples/sum.js && git commit -qm "Moves sum.js to examples/"

printf 'Москва\r\nТампере\r\n' > города.txt && git add города.txt && git commit -qm "Adds the city list"

printf 'Москва\nТампере\nТурку\n' > города.txt && git add города.txt

git diff --cached --name-only

"\320\263\320\276\321\200\320\276\320\264\320\260.txt"

tools/old-check.sh --staged

check-diff-artifacts: clean

tools/check-diff-artifacts.sh --staged

ARTIFACT: города.txt [staged]: CRLF line endings lost (base CRLF on all 2 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md

git commit -qm "Adds Turku"

Last, a CRLF loss with an edit that is not staged:

sed -i 's/\r$//; s/two numbers/two numbers a and b/' examples/sum.js

tools/old-check.sh --all

check-diff-artifacts: clean

tools/check-diff-artifacts.sh --all

ARTIFACT: examples/sum.js [all]: CRLF line endings lost (base CRLF on all 4 lines, now LF) together with other changes; a conversion alone passes check-diff-artifacts: 1 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md

The fix: block only what is never intended

The two problems have one cause: rules 1 and 4 measure the shape of a diff, and both damage and ordinary work can have that shape. The new version measures the damage itself. Each rule compares the bytes of the new side with the bytes of the base side, and blocks only what nobody ever does on purpose in this repository. What deserves attention but is not damage becomes a NOTE that does not block.

The line-ending rule shows the idea best. Losing CRLF is damage when it hides an edit, because a diff where every line changed cannot be reviewed. A conversion with nothing else hides nothing, and that can be proved: the new file must be exactly the base with the CR before each line break removed. The new version checks this with one comparison and lets such a file through with a note, so the repository can move to LF in commits that only convert. On the scratch repository, the old version blocks a pure conversion twice and the new one passes it:

git checkout -- examples/sum.js

sed -i 's/\r$//' examples/sum.js && git add examples/sum.js

tools/old-check.sh --staged

ARTIFACT: examples/sum.js [staged]: whitespace/EOL-only diff (4+/ 4-, invisible under -b) ARTIFACT: examples/sum.js [staged]: CRLF line endings lost (HEAD blob has CR)

tools/check-diff-artifacts.sh --staged

NOTE: examples/sum.js [staged]: converted from CRLF to LF, nothing else changed (4 lines) check-diff-artifacts: clean (1 note(s), not blocking)

A conversion mixed with an edit, as in all three cases above, still blocks, and the message says what would pass. The exception is one way: LF to CRLF always blocks, because the goal is a repository that ends up LF.

Version 0.0.4 and version 0.2.0
0.0.40.2.0
whitespace or line-ending only diff: blocksa line-ending change is judged by the line-ending rule; any other whitespace-only diff is a NOTE
CRLF lost: every CR gonethe ending style of each side (lf, crlf, mixed): CRLF lost together with other changes, mixed endings in a file that had one style, LF flipped to CRLF; a pure conversion to LF is a NOTE
U+FFFD addedkept for existing files; a new file with U+FFFD is a NOTE
full-file rewrite: blocksremoved; mojibake, UTF-8 validity and BOM rules check for the damage it stood for
renames and Cyrillic paths skippedgit diff --raw -z -M: renames compared with their source, paths unquoted
--all = --staged--all compares HEAD with the working tree; new --commit REV

How each rule works is in How the checks work. The --commit mode was added first, because it is what makes the next test possible.

Testing on the whole history

Unit tests check the cases someone thought of. The history of the repository holds the cases that really happened: every commit is a staged diff that the check saw, or would have seen. The test is to run both versions on every commit and read every finding. The new version does it with --commit; a small wrapper prefixes each finding with the commit id:

vi replay.sh

#!/usr/bin/env bash # replay.sh REV: run the check on one commit, prefix each finding with REV out=$(container/tools/check-diff-artifacts.sh --commit "$1" 2>&1) grep -E '^(ARTIFACT|NOTE):' <<< "$out" | sed "s/^/$1 /"

chmod +x replay.sh

git rev-list --no-merges HEAD > commits.txt

wc -l < commits.txt

1174

xargs -P 8 runs eight commits at a time; one commit per call (-n 1):

time (xargs -P 8 -n 1 ./replay.sh < commits.txt > hits.txt)

real 16m15,641s user 11m58,344s sys 21m46,361s

Version 0.0.4 had no way to check a commit, so for the old rules a second wrapper ran the same four tests on the pair REV^ and REV with git diff REV^ REV and git show REV:path.

Results of the replay

The commits that the new version blocks:

awk '$2 == "ARTIFACT:" {print substr($1, 1, 9)}' hits.txt | sort | uniq -c | sort -rn

275 2af4d0554 6 45e2c4d58 4 8d889c516 1 b68ebc079 1 a402d2992

Every one of them was read:

None of the 14 false positives is among them. The notes were read too:

awk '$2 == "NOTE:"' hits.txt | sed -E 's/^[0-9a-f]+ NOTE: [^[]*\[[^]]*\]: //; s/(^|[ (/])[0-9]+/\1N/g' | sort | uniq -c | sort -rn

53 converted from CRLF to LF, nothing else changed (N lines) 16 whitespace-only change (N+/N-: indentation, trailing spaces), not blocking 8 new file with mixed line endings (N of N lines CRLF) 7 new file with U+FFFD on N line(s) 4 converted from mixed endings to LF, nothing else changed (N of N lines had CRLF) 2 new file with a UTF-8 BOM

Two of these kinds were blocks in the first draft of the new version and came out of the replay as wrong. The first commit of the repository, an import of 9,155 files, has pages that show U+FFFD on purpose (a page about chr(), output of /dev/urandom), and a vendored library ships READMEs with a BOM. A new file has no base to be damaged against, so for new files these are now notes, except a BOM in a new PHP file, which breaks the headers of the page.

Calibrating the mojibake rule

Mojibake is UTF-8 read in a one-byte encoding and saved again. The rule counts byte patterns of such text: a UTF-8 lead byte read as cp1252 or latin-1, and the Cyrillic lead bytes read as cp1251, which turn every Russian letter into Р or С followed by a character that Russian does not put there. The first version of the pattern was run on every tracked text file of the repository, where no mojibake was expected. It found genuine text: a mathematics dictionary writes "category С°" and "С…" before an ellipsis. Those pairs went out of the pattern. With the final pattern of the script in the variable MOJIBAKE, the search finds three files:

LC_ALL=C git grep -c -I -P "$MOJIBAKE"

container/src/dev/python/collections/string/index.php:1 container/src/dev/python/strings.php:1 container/src/ru/code/python/objects/str/index.php:1

All three are Python lessons that quote one mojibake word to show what a wrong decode looks like. Such a quote is 2 to 6 sequences, and one damaged line of Russian is 20 or more, so the rule needs ten new sequences in a file. The quotes pass, a damaged page does not.

The check on its own change

The first run of the new version over the working tree found three artifacts, all in the change itself:

ARTIFACT: container/tests/linters/test_check_diff_artifacts_sh.py [all]: U+FFFD replacement chars introduced (0 -> 1, a lossy decode?) ARTIFACT: container/tools/CHECK_DIFF_ARTIFACTS.md [all]: mojibake introduced (0 -> 6 sequences like é or Рё: UTF-8 read as cp1252, latin-1 or cp1251 and saved again) ARTIFACT: container/tools/check-diff-artifacts.sh [all]: mojibake introduced (0 -> 7 sequences like é or Рё: UTF-8 read as cp1252, latin-1 or cp1251 and saved again) check-diff-artifacts: 3 artifact(s). Fix them, or audit and skip this hook only: container/tools/CHECK_DIFF_ARTIFACTS.md

The test file had a literal U+FFFD in a test case, and the documentation and the script quoted mojibake as examples. A check that cannot commit itself would have been skipped on its first day. The fix: the script is ASCII only, the documentation describes the examples instead of quoting them, the tests build the damaged bytes at run time ("\N{REPLACEMENT CHARACTER}", .encode().decode("cp1251").encode()), and the threshold of ten covers a short quote. This page shows its mojibake samples as numeric HTML entities, so its source passes the check too.

Tests in scratch repositories

The self-test of the script grew from 5 cases to 36. Each one creates a git repository in a temporary folder, commits a base file, stages a new version and runs the real script:

The list with the output of the run is in Tests.

Speed

A pre-commit check runs on every commit, so its time is paid every time. The first draft of the new version started about twenty processes per file: two git diff --no-index calls, line counts, greps and iconv. Three changes cut that down: one git diff --numstat and one git diff -b --numstat for all files instead of two calls per file, no line counting when neither side has a CR, and no encoding checks when the new side is plain ASCII. Measured with --commit on the same machine under the same load:

Time per commit
CommitFirst draftFinal
48 files8.7 s1.7 s
17 files3.7 s2.0 s
382 files63.5 s21.0 s

A linter that checked nothing

Before the commit, all hooks were run on the changed files with pre-commit run --files, and ShellCheck passed. The commit itself failed:

shellcheck (maintained .sh)..............................................Failed - hook id: shellcheck - exit code: 1 linting 1 shell script(s) with shellcheck ... In container/tools/check-diff-artifacts.sh line 258: into["$path"]="$adds $dels" ^-----------^ SC2034 (warning): into appears unused. Verify use (or export if used externally).

The shellcheck hook of the repository is configured with pass_filenames: false and lints the staged scripts itself. Nothing was staged during --files, so it linted nothing and passed. The warning is a known false alarm for a bash nameref (local -n into=...), and the function was rewritten without one. The real check was the hook's own linter on the file, together with the same linter on the old staged copy, which had to fail with SC2034 to prove that the file was really read.

What we learned

Article author: Klavdiya Kondakova

Banner image