A pre-commit check that blocked a correct commit: git diff -w and git diff -b

This page is about the first version of the check, 0.0.4, and the outputs on it come from that version. The check was later rebuilt to block only line-ending and encoding damage and to let whitespace-only edits, 1:1 rewrites and pure conversions to LF through. The current version is described in A pre-commit check for line-ending and encoding damage, and why and how it changed in Reworking a pre-commit check so it fires only on real damage. The difference between git diff -w and git diff -b explained here still holds: the current version uses -b for its whitespace note.

A pre-commit hook that blocks good commits is worse than none: people learn to bypass it. This page is the story of one such hook, a check for accidental whitespace-only diffs, that refused a commit whose only change was the spelling of a word. It shows what the check was, why it fired, the one-word fix and the test that keeps it fixed. Every command was run in a fresh repository with git 2.43.0 and bash 5.2; the output is pasted as it printed.

The check and what it guards

The site repository has a script, check-diff-artifacts.sh, that runs from the pre-commit framework as a local hook on every commit. It looks at the staged diff and refuses four kinds of damage that scripted bulk edits leave behind:

The first check is the one that went wrong. Its idea is simple: a diff that exists, but disappears when whitespace is ignored, contains no real change. In bash it compares the line counts of two git diff calls:

adds, dels = lines added and deleted by: git diff --cached --numstat wsum = lines changed when whitespace is ignored: git diff --cached -w --numstat if wsum is empty, the diff was whitespace only: report it

The commit it blocked

A task fixed the spelling of JavaScript across the site: the pages said "Java Script" in titles, keywords and descriptions. Six files were staged, and the commit failed. Three of the six report lines are shown:

check-diff-artifacts (staged)...........................................Failed - hook id: check-diff-artifacts - exit code: 1 ARTIFACT: container/src/dev/js/array_sort.php [staged]: whitespace/EOL-only diff (2+/ 2-, invisible under -w) ARTIFACT: container/src/dev/js/tictactoe2.php [staged]: whitespace/EOL-only diff (2+/ 2-, invisible under -w) ARTIFACT: container/src/ru/code/js/ibs_to_kg.php [staged]: whitespace/EOL-only diff (3+/ 3-, invisible under -w)

The diffs were real. A line such as title: 'Java Script Promise' became title: 'JavaScript Promise'. Nothing was wrong with the commit and the hook was the problem, so the commit stayed uncommitted until the hook was fixed. --no-verify would have skipped the hook for this one commit, but a hook that has to be skipped for correct work is a hook with a bug, and the bug stays for the next person.

Why: git diff -w

-w is short for --ignore-all-space: git compares the lines as if every space and tab were deleted from them. That is exactly the right model for a CRLF flip or a re-indent. It is the wrong model for the spelling fix, because Java Script and JavaScript are the same text once the space is gone. The effect can be reproduced in a scratch repository with three lines in a file:

git init -q && git config user.email a@b && git config user.name n

printf "<?php\n\$page_keywords = 'Java Script, Java Script basics';\n title: 'Java Script Promise',\n" > page.php

git add page.php && git commit -qm init

sed -i "s/Java Script/JavaScript/g" page.php && git add page.php

git diff --cached --numstat

2 2 page.php

git diff --cached -w --numstat

The second command prints nothing: with -w git sees no difference at all. The script took the empty answer for "whitespace only":

git show HEAD:container/tools/check-diff-artifacts.sh > /tmp/old-check-diff-artifacts.sh && chmod +x /tmp/old-check-diff-artifacts.sh

/tmp/old-check-diff-artifacts.sh --staged

ARTIFACT: page.php [staged]: whitespace/EOL-only diff (2+/ 2-, invisible under -w)

The check had no way to tell a space that moved or vanished at the edge of a line from a space that was added or removed between two letters. The second kind changes words, which is the opposite of an artifact.

The fix: git diff -b

-b, or --ignore-space-change, is the milder option. It ignores whitespace at the end of a line and treats every run of spaces and tabs as one equal separator, but it does not delete the separator: "Java Script" and "JavaScript" still differ. The change in the script is one flag and a comment:

- wsum=$(git diff "${diff_args[@]}" -w --numstat -- "$f" | awk '{print $1+$2}') + # -b, not -w: -w also hides a space added or removed inside a line + # ("Java Script" -> "JavaScript"), which is a real text change. + wsum=$(git diff "${diff_args[@]}" -b --numstat -- "$f" | awk '{print $1+$2}')

The message now says invisible under -b, and the version header of the script went up by one. On the same staged spelling fix, -b still sees the two changed lines and the hook passes:

git diff --cached -b --numstat

2 2 page.php

container/tools/check-diff-artifacts.sh --staged

check-diff-artifacts: clean

What the check exists for must still be caught. After committing the fix, the file is flipped to CRLF endings and staged:

git commit -qm fix && sed -i "s/\$/\r/" page.php && git add page.php

git diff --cached --numstat

3 3 page.php

git diff --cached -b --numstat

container/tools/check-diff-artifacts.sh --staged

ARTIFACT: page.php [staged]: whitespace/EOL-only diff (3+/ 3-, invisible under -b)

Under -b the carriage returns at the end of the lines are ignored, so the diff is empty and the hook reports it, with a non-zero exit status. This is how the two flags compare on the cases the check cares about:

What each flag reports as whitespace only
Change-w-b
CRLF to LF flipwhitespace onlywhitespace only
spaces added at the end of a linewhitespace onlywhitespace only
tabs replaced by spaces in the indentwhitespace onlywhitespace only
one space replaced by threewhitespace onlywhitespace only
Java Script to JavaScriptwhitespace only (wrong)a real change
indent added to a line that had nonewhitespace onlya real change

A test for the hook

The bug was in a script nobody tested, so the fix comes with a test. It follows the same pattern as the tests of the other pre-commit gates: a scratch git repository per test, the real script run in it, the exit status and the text of the report checked. The repository holds one committed file; each case stages a changed version of it and runs --staged:

ORIGINAL = b"a Java Script b\n x = 1\nfoo\n" @pytest.mark.parametrize( "content", [ b"a Java Script b\r\n x = 1\r\nfoo\r\n", b"a Java Script b \n x = 1\nfoo\n", b"a Java Script b\n\tx = 1\nfoo\n", b"a Java Script b\n x = 1\nfoo\n", ], ids=["crlf", "trailing", "tab-indent", "space-run"], ) def test_whitespace_only_diff_is_flagged(repo, content): result = check(repo, content) assert result.returncode == 1 assert "whitespace/EOL-only diff" in result.stdout + result.stderr def test_space_removed_inside_a_line_passes(repo): result = check(repo, b"a JavaScript b\n x = 1\nfoo\n") assert "whitespace/EOL-only diff" not in result.stdout + result.stderr assert result.returncode == 0

python -m pytest tests/linters/test_check_diff_artifacts_sh.py -v

tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[crlf] PASSED [ 20%] tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[trailing] PASSED [ 40%] tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[tab-indent] PASSED [ 60%] tests/linters/test_check_diff_artifacts_sh.py::test_whitespace_only_diff_is_flagged[space-run] PASSED [ 80%] tests/linters/test_check_diff_artifacts_sh.py::test_space_removed_inside_a_line_passes PASSED [100%] ============================== 5 passed in 0.42s ===============================

Four cases keep the original purpose of the hook and one keeps the bug from coming back. With the old -w the fifth test fails.

Limits and what not to do

More on writing hooks that look only at what is staged is in linting what is staged, and on the order of hooks that write files in pre-commit and post-commit together.

Article author: Klavdiya Kondakova

Banner image