Testing a shell pipeline under a file lock
Introduction
The prod-log pipeline of the site is one script with five steps: fetch the
prod logs, fetch the device and user profiles, rebuild
users.db, rebuild botlist.csv,
rebuild metrika.db. cron
runs it every thirty minutes, and the backoffice can start the same script
by hand.
Two runs at once would fetch the same files twice and write the same
database from two processes. The script therefore takes a lock in the log
directory before its first step and refuses to start when the lock is
taken. Its test, container/tests/tools/test_refresh_metrika.py,
runs the real script twice: once with a free lock, once while another
process already holds it. Every step is skipped by a flag, so the tests
fetch nothing and build nothing.
Tools used here: Python , pytest , Bash .
The lock in the script
Three shell lines do the work. The script opens the lock file on file descriptor 9 for the whole run, asks flock for a non-blocking exclusive lock on that descriptor, and exits 0 when the lock is already held. The descriptor is not closed anywhere, so the lock lives as long as the process.
The script container/tools/analyze_logs/refresh-metrika.sh (abridged):
#!/usr/bin/env bash set -euo pipefail _START_DIR="$(pwd -P)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SKIP_FETCH=0 SKIP_DEVICES=0 SKIP_USERS=0 SKIP_BOTS=0 SKIP_DB=0 while [ $# -gt 0 ]; do case "$1" in --skip-fetch) SKIP_FETCH=1; shift ;; --skip-devices) SKIP_DEVICES=1; shift ;; --skip-users) SKIP_USERS=1; shift ;; --skip-bots) SKIP_BOTS=1; shift ;; --skip-db) SKIP_DB=1; shift ;; ... esac done . "$SCRIPT_DIR/log-root.sh" mkdir -p "$AREDEL_LOG_DIR" LOCK="$AREDEL_LOG_DIR/refresh-metrika.lock" exec 9>"$LOCK" if ! flock -n 9; then echo "[$(date '+%F %T')] refresh-metrika: another run holds $LOCK, skipping" exit 0 fi echo "[$(date '+%F %T')] refresh-metrika: start" # steps 1-5, each guarded by its --skip-* flag echo "[$(date '+%F %T')] refresh-metrika: done"
flock -n does not wait: it returns a non-zero status
at once when another process holds the lock, and
if ! turns that into the skip branch. The exit code
of the skip branch is 0 on purpose. A second run is not an error, it is a
normal event when a slow run overlaps the next cron tick, and a non-zero
exit would fill the cron log and the backoffice with failures that need no
attention.
The lock file is created by the redirection itself, before
flock is asked anything, so a cancelled run leaves
an empty file behind and nothing else. The path comes from
log-root.sh, the shell half of the resolver that the
log paths
page tests.
The test file
Forty-one lines: imports, the path of the script, the list of skip flags, a module-level skipif, one helper and two tests. The two module-level names find the script and switch every step off. The whole module is skipped when the machine has no flock, because the shell command comes from util-linux and a minimal image may not have it.
from __future__ import annotations import os import shutil import subprocess from pathlib import Path import pytest SCRIPT = Path(__file__).resolve().parents[2] / "tools" / "analyze_logs" / "refresh-metrika.sh" SKIP_ALL = ["--skip-fetch", "--skip-devices", "--skip-users", "--skip-bots", "--skip-db"] pytestmark = pytest.mark.skipif(shutil.which("flock") is None, reason="flock (util-linux) is not installed") def _run(log_dir: Path, *prefix: str) -> subprocess.CompletedProcess[str]: env = {**os.environ, "AREDEL_LOG_DIR": str(log_dir)} return subprocess.run([*prefix, str(SCRIPT), *SKIP_ALL], env=env, capture_output=True, text=True, timeout=60, check=False)
parents[2] walks from the test file to
container: the test sits in
container/tests/tools/, so the first parent is
tools, the second is
tests, the third is the project root of the suite.
_run() builds the command as a list: an optional
prefix, the script, then the five skip flags. The prefix is what the second
test uses to hold the lock from the outside; in the first test it is empty.
env={**os.environ, ...} keeps
PATH, without which the shebang cannot find
bash, and replaces only
AREDEL_LOG_DIR, so the lock lands in the temporary
directory of the test.
The arguments of
subprocess.run
matter here as much as in any test:
capture_output=True keeps stdout and stderr out of
the test log and inside the result object,
text=True makes them strings,
timeout=60 stops a script that waits for a lock
forever, and check=False is deliberate: an exit code
is a result to assert, not an exception to raise.
A free lock runs the chain
def test_free_lock_runs_the_chain(tmp_path: Path) -> None: result = _run(tmp_path) assert result.returncode == 0, result.stderr assert "refresh-metrika: done" in result.stdout assert (tmp_path / "refresh-metrika.lock").is_file()
The empty log directory is the whole setup. Three assertions then describe
the run from three sides: the script exited cleanly, it reached its last
line, and it left the lock file where the next run will look for it. The
first assertion prints the stderr of the script when it fails, which is how
an error inside the shell code shows up in the pytest output instead of an
empty mismatch.
The middle assertion is a substring, not the whole line, because the line
starts with a timestamp. A test that compared the whole line would fail
every second.
A held lock skips the run
def test_held_lock_skips_the_run(tmp_path: Path) -> None: lock = tmp_path / "refresh-metrika.lock" result = _run(tmp_path, "flock", str(lock)) assert result.returncode == 0, result.stderr assert "another run holds" in result.stdout assert "refresh-metrika: start" not in result.stdout
The prefix turns the command into
flock /tmp/.../refresh-metrika.lock /path/to/refresh-metrika.sh ....
The outer flock opens the file, takes an exclusive
lock on it and only then starts the script, which opens the same file on
its own descriptor 9. Locks belong to the open file, not to the name, so
the second request conflicts with the first and returns immediately.
The third assertion is the one that catches a half-fixed script. A run that
prints the "another run holds" line and then continues would pass
the first two assertions: it exits 0 and it mentions the lock. Only the
absence of refresh-metrika: start proves that the
chain ran no step.
Run the file
Run the file from the container directory:
./tools/run-tests-venv.sh -- tests/tools/test_refresh_metrika.py -q
Checked 5 packages in 3ms BASE_URL=http://127.0.0.1:8080 SITEMAP_FILE=src/sitemap.xml .. [100%] 2 passed in 1.20s
The first line is uv checking the packages of the venv, the second the two variables the script exports for the tests; then come the two dots. On a machine without flock the same command prints two s marks and 2 skipped instead, with the reason from pytestmark. That is the difference between a test that cannot run and a test that fails.
Article author: Arthur Isaev