Testing a shell pipeline under a file lock

Introduction

The prod-log pipeline of the site is one script with five steps: fetch the prod logs, fetch the device and user profiles, rebuild users.db, rebuild botlist.csv, rebuild metrika.db. cron runs it every thirty minutes, and the backoffice can start the same script by hand.

Two runs at once would fetch the same files twice and write the same database from two processes. The script therefore takes a lock in the log directory before its first step and refuses to start when the lock is taken. Its test, container/tests/tools/test_refresh_metrika.py, runs the real script twice: once with a free lock, once while another process already holds it. Every step is skipped by a flag, so the tests fetch nothing and build nothing.

Tools used here: Python , pytest , Bash .

The lock in the script

Three shell lines do the work. The script opens the lock file on file descriptor 9 for the whole run, asks flock for a non-blocking exclusive lock on that descriptor, and exits 0 when the lock is already held. The descriptor is not closed anywhere, so the lock lives as long as the process.

The script container/tools/analyze_logs/refresh-metrika.sh (abridged):

#!/usr/bin/env bash set -euo pipefail _START_DIR="$(pwd -P)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SKIP_FETCH=0 SKIP_DEVICES=0 SKIP_USERS=0 SKIP_BOTS=0 SKIP_DB=0 while [ $# -gt 0 ]; do case "$1" in --skip-fetch) SKIP_FETCH=1; shift ;; --skip-devices) SKIP_DEVICES=1; shift ;; --skip-users) SKIP_USERS=1; shift ;; --skip-bots) SKIP_BOTS=1; shift ;; --skip-db) SKIP_DB=1; shift ;; ... esac done . "$SCRIPT_DIR/log-root.sh" mkdir -p "$AREDEL_LOG_DIR" LOCK="$AREDEL_LOG_DIR/refresh-metrika.lock" exec 9>"$LOCK" if ! flock -n 9; then echo "[$(date '+%F %T')] refresh-metrika: another run holds $LOCK, skipping" exit 0 fi echo "[$(date '+%F %T')] refresh-metrika: start" # steps 1-5, each guarded by its --skip-* flag echo "[$(date '+%F %T')] refresh-metrika: done"

flock -n does not wait: it returns a non-zero status at once when another process holds the lock, and if ! turns that into the skip branch. The exit code of the skip branch is 0 on purpose. A second run is not an error, it is a normal event when a slow run overlaps the next cron tick, and a non-zero exit would fill the cron log and the backoffice with failures that need no attention.

The lock file is created by the redirection itself, before flock is asked anything, so a cancelled run leaves an empty file behind and nothing else. The path comes from log-root.sh, the shell half of the resolver that the log paths page tests.

The test file

Forty-one lines: imports, the path of the script, the list of skip flags, a module-level skipif, one helper and two tests. The two module-level names find the script and switch every step off. The whole module is skipped when the machine has no flock, because the shell command comes from util-linux and a minimal image may not have it.

from __future__ import annotations import os import shutil import subprocess from pathlib import Path import pytest SCRIPT = Path(__file__).resolve().parents[2] / "tools" / "analyze_logs" / "refresh-metrika.sh" SKIP_ALL = ["--skip-fetch", "--skip-devices", "--skip-users", "--skip-bots", "--skip-db"] pytestmark = pytest.mark.skipif(shutil.which("flock") is None, reason="flock (util-linux) is not installed") def _run(log_dir: Path, *prefix: str) -> subprocess.CompletedProcess[str]: env = {**os.environ, "AREDEL_LOG_DIR": str(log_dir)} return subprocess.run([*prefix, str(SCRIPT), *SKIP_ALL], env=env, capture_output=True, text=True, timeout=60, check=False)

parents[2] walks from the test file to container: the test sits in container/tests/tools/, so the first parent is tools, the second is tests, the third is the project root of the suite.

_run() builds the command as a list: an optional prefix, the script, then the five skip flags. The prefix is what the second test uses to hold the lock from the outside; in the first test it is empty. env={**os.environ, ...} keeps PATH, without which the shebang cannot find bash, and replaces only AREDEL_LOG_DIR, so the lock lands in the temporary directory of the test.

The arguments of subprocess.run matter here as much as in any test: capture_output=True keeps stdout and stderr out of the test log and inside the result object, text=True makes them strings, timeout=60 stops a script that waits for a lock forever, and check=False is deliberate: an exit code is a result to assert, not an exception to raise.

A free lock runs the chain

def test_free_lock_runs_the_chain(tmp_path: Path) -> None: result = _run(tmp_path) assert result.returncode == 0, result.stderr assert "refresh-metrika: done" in result.stdout assert (tmp_path / "refresh-metrika.lock").is_file()

The empty log directory is the whole setup. Three assertions then describe the run from three sides: the script exited cleanly, it reached its last line, and it left the lock file where the next run will look for it. The first assertion prints the stderr of the script when it fails, which is how an error inside the shell code shows up in the pytest output instead of an empty mismatch.

The middle assertion is a substring, not the whole line, because the line starts with a timestamp. A test that compared the whole line would fail every second.

A held lock skips the run

def test_held_lock_skips_the_run(tmp_path: Path) -> None: lock = tmp_path / "refresh-metrika.lock" result = _run(tmp_path, "flock", str(lock)) assert result.returncode == 0, result.stderr assert "another run holds" in result.stdout assert "refresh-metrika: start" not in result.stdout

The prefix turns the command into flock /tmp/.../refresh-metrika.lock /path/to/refresh-metrika.sh .... The outer flock opens the file, takes an exclusive lock on it and only then starts the script, which opens the same file on its own descriptor 9. Locks belong to the open file, not to the name, so the second request conflicts with the first and returns immediately.

The third assertion is the one that catches a half-fixed script. A run that prints the "another run holds" line and then continues would pass the first two assertions: it exits 0 and it mentions the lock. Only the absence of refresh-metrika: start proves that the chain ran no step.

Run the file

Run the file from the container directory:

./tools/run-tests-venv.sh -- tests/tools/test_refresh_metrika.py -q

Checked 5 packages in 3ms BASE_URL=http://127.0.0.1:8080 SITEMAP_FILE=src/sitemap.xml .. [100%] 2 passed in 1.20s

The first line is uv checking the packages of the venv, the second the two variables the script exports for the tests; then come the two dots. On a machine without flock the same command prints two s marks and 2 skipped instead, with the reason from pytestmark. That is the difference between a test that cannot run and a test that fails.

Article author: Arthur Isaev

Banner image