A pre-commit check that blocked a correct commit: git diff -w and git diff -b
This page is about the first version of the check, 0.0.4, and the outputs on it come from that version. The check was later rebuilt to block only line-ending and encoding damage and to let whitespace-only edits, 1:1 rewrites and pure conversions to LF through. The current version is described in A pre-commit check for line-ending and encoding damage, and why and how it changed in Reworking a pre-commit check so it fires only on real damage. The difference between git diff -w and git diff -b explained here still holds: the current version uses -b for its whitespace note.
A pre-commit hook that blocks good commits is worse than none: people learn to bypass it. This page is the story of one such hook, a check for accidental whitespace-only diffs, that refused a commit whose only change was the spelling of a word. It shows what the check was, why it fired, the one-word fix and the test that keeps it fixed. Every command was run in a fresh repository with git 2.43.0 and bash 5.2; the output is pasted as it printed.
The check and what it guards
The site repository has a script, check-diff-artifacts.sh, that runs from the pre-commit framework as a local hook on every commit. It looks at the staged diff and refuses four kinds of damage that scripted bulk edits leave behind:
- a diff that changes only whitespace or line endings, for example every line of a file flipped from CRLF to LF;
- CRLF line endings lost in a file that had them in HEAD;
- U+FFFD replacement characters, the sign of an encoding round trip;
- a full-file rewrite: as many lines added as deleted in a file longer than 100 lines.
The first check is the one that went wrong. Its idea is simple: a diff that exists, but disappears when whitespace is ignored, contains no real change. In bash it compares the line counts of two git diff calls:
adds, dels = lines added and deleted by: git diff --cached --numstat wsum = lines changed when whitespace is ignored: git diff --cached -w --numstat if wsum is empty, the diff was whitespace only: report it
The commit it blocked
A task fixed the spelling of JavaScript across the site: the pages said "Java Script" in titles, keywords and descriptions. Six files were staged, and the commit failed. Three of the six report lines are shown:
check-diff-artifacts (staged)...........................................Failed - hook id: check-diff-artifacts - exit code: 1 ARTIFACT: container/src/dev/js/array_sort.php [staged]: whitespace/EOL-only diff (2+/ 2-, invisible under -w) ARTIFACT: container/src/dev/js/tictactoe2.php [staged]: whitespace/EOL-only diff (2+/ 2-, invisible under -w) ARTIFACT: container/src/ru/code/js/ibs_to_kg.php [staged]: whitespace/EOL-only diff (3+/ 3-, invisible under -w)
The diffs were real. A line such as title: 'Java Script Promise' became title: 'JavaScript Promise'. Nothing was wrong with the commit and the hook was the problem, so the commit stayed uncommitted until the hook was fixed. --no-verify would have skipped the hook for this one commit, but a hook that has to be skipped for correct work is a hook with a bug, and the bug stays for the next person.
Why: git diff -w
-w is short for --ignore-all-space: git compares the lines as if every space and tab were deleted from them. That is exactly the right model for a CRLF flip or a re-indent. It is the wrong model for the spelling fix, because Java Script and JavaScript are the same text once the space is gone. The effect can be reproduced in a scratch repository with three lines in a file:
git init -q && git config user.email a@b && git config user.name n
printf "<?php\n\$page_keywords = 'Java Script, Java Script basics';\n title: 'Java Script Promise',\n" > page.php
git add page.php && git commit -qm init
sed -i "s/Java Script/JavaScript/g" page.php && git add page.php
git diff --cached --numstat
2 2 page.php
git diff --cached -w --numstat
The second command prints nothing: with -w git sees no difference at all. The script took the empty answer for "whitespace only":
git show HEAD:container/tools/check-diff-artifacts.sh > /tmp/old-check-diff-artifacts.sh && chmod +x /tmp/old-check-diff-artifacts.sh
/tmp/old-check-diff-artifacts.sh --staged
ARTIFACT: page.php [staged]: whitespace/EOL-only diff (2+/ 2-, invisible under -w)
The check had no way to tell a space that moved or vanished at the edge of a line from a space that was added or removed between two letters. The second kind changes words, which is the opposite of an artifact.
The fix: git diff -b
-b, or --ignore-space-change, is the milder option. It ignores whitespace at the end of a line and treats every run of spaces and tabs as one equal separator, but it does not delete the separator: "Java Script" and "JavaScript" still differ. The change in the script is one flag and a comment:
- wsum=$(git diff "${diff_args[@]}" -w --numstat -- "$f" | awk '{print $1+$2}') + # -b, not -w: -w also hides a space added or removed inside a line + # ("Java Script" -> "JavaScript"), which is a real text change. + wsum=$(git diff "${diff_args[@]}" -b --numstat -- "$f" | awk '{print $1+$2}')
The message now says invisible under -b, and the version header of the script went up by one. On the same staged spelling fix, -b still sees the two changed lines and the hook passes:
git diff --cached -b --numstat
2 2 page.php
container/tools/check-diff-artifacts.sh --staged
check-diff-artifacts: clean
What the check exists for must still be caught. After committing the fix, the file is flipped to CRLF endings and staged:
git commit -qm fix && sed -i "s/\$/\r/" page.php && git add page.php
git diff --cached --numstat
3 3 page.php
git diff --cached -b --numstat
container/tools/check-diff-artifacts.sh --staged
ARTIFACT: page.php [staged]: whitespace/EOL-only diff (3+/ 3-, invisible under -b)
Under -b the carriage returns at the end of the lines are ignored, so the diff is empty and the hook reports it, with a non-zero exit status. This is how the two flags compare on the cases the check cares about:
| Change | -w | -b |
|---|---|---|
| CRLF to LF flip | whitespace only | whitespace only |
| spaces added at the end of a line | whitespace only | whitespace only |
| tabs replaced by spaces in the indent | whitespace only | whitespace only |
| one space replaced by three | whitespace only | whitespace only |
| Java Script to JavaScript | whitespace only (wrong) | a real change |
| indent added to a line that had none | whitespace only | a real change |
A test for the hook
The bug was in a script nobody tested, so the fix comes with a test. It follows the same pattern as the tests of the other pre-commit gates: a scratch git repository per test, the real script run in it, the exit status and the text of the report checked. The repository holds one committed file; each case stages a changed version of it and runs --staged:
ORIGINAL = b"a Java Script b\n x = 1\nfoo\n" @pytest.mark.parametrize( "content", [ b"a Java Script b\r\n x = 1\r\nfoo\r\n", b"a Java Script b \n x = 1\nfoo\n", b"a Java Script b\n\tx = 1\nfoo\n", b"a Java Script b\n x = 1\nfoo\n", ], ids=["crlf", "trailing", "tab-indent", "space-run"], ) def test_whitespace_only_diff_is_flagged(repo, content): result = check(repo, content) assert result.returncode == 1 assert "whitespace/EOL-only diff" in result.stdout + result.stderr def test_space_removed_inside_a_line_passes(repo): result = check(repo, b"a JavaScript b\n x = 1\nfoo\n") assert "whitespace/EOL-only diff" not in result.stdout + result.stderr assert result.returncode == 0
python -m pytest tests/linters/test_check_diff_artifacts_sh.py -v
tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[crlf] PASSED [ 20%] tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[trailing] PASSED [ 40%] tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[tab-indent] PASSED [ 60%] tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[space-run] PASSED [ 80%] tests/linters/test_check_diff_artifacts_sh.py::test_space_removed_inside_a_line_passes PASSED [100%] ============================== 5 passed in 0.42s ===============================
Four cases keep the original purpose of the hook and one keeps the bug from coming back. With the old -w the fifth test fails.
Limits and what not to do
- Added indentation is not flagged any more. A line that had no leading whitespace and got four spaces is a change under -b. Such a diff is rare in scripted edits and it is a real change to the file, so the check accepts it. If it must be caught, add a second comparison with --ignore-space-at-eol --ignore-cr-at-eol, which ignores only the end of the line.
- Do not rewrite the text to get past the hook. Changing a nearby word so that the diff is not space-only commits a change nobody wanted.
- Do not use --no-verify as the fix. It skips the hook for one commit and leaves the bug in place. Fix the hook, commit the fix first, then commit the work that it blocked.
- Test the check with the case that failed. The false positive came from a real commit; the first test of the fix is that same commit.
More on writing hooks that look only at what is staged is in linting what is staged, and on the order of hooks that write files in pre-commit and post-commit together.
Article author: Klavdiya Kondakova