Testing the error-log analyzer: dates, --since and --check-prod

Introduction

analyze_errors.py reads the Apache error logs of prod, groups identical errors and answers four questions per group: is the address a real page or a wrong one, is the failure reproducible from the local src tree, when was it first and last seen, and what does prod answer for that page right now.

Its test, container/tests/tools/test_analyze_errors.py, has four tests. Each one takes a part of that pipeline: the timestamp parser, the report lines with first and last seen, the --since filter that drops everything older than a deploy, and the --check-prod option with its cap. There is no log directory, no network and no PHP : the log lines are written by the test, and the client addresses are from the documentation range 203.0.113.0/24.

Tools used here: Python , pytest .

Load the module by path

The analyzer is a script in container/tools/analyze_logs, not an installed package, and it imports its neighbour log_paths.py. A plain import would fail twice: the folder is not on sys.path, and the module is not on any path at all. The test loads it from its file and adds the folder to the import path only while the module body runs.

The test file container/tests/tools/test_analyze_errors.py, header and loader:

from __future__ import annotations import importlib.util import sys from datetime import datetime from pathlib import Path ANALYZE_LOGS = Path(__file__).resolve().parents[2] / "tools" / "analyze_logs" PREFIX = "/home/a/andreyv/aredel.com/public_html/" def _module(): sys.path.insert(0, str(ANALYZE_LOGS)) try: spec = importlib.util.spec_from_file_location("aredel_analyze_errors", ANALYZE_LOGS / "analyze_errors.py") assert spec and spec.loader module = importlib.util.module_from_spec(spec) sys.modules[spec.name] = module # dataclasses look their module up here spec.loader.exec_module(module) return module finally: sys.path.remove(str(ANALYZE_LOGS))

The importlib recipe is the usual three calls: build a spec from the path, build a module object from the spec, run the file in that object. Two details belong to this module in particular:

The loader is called by every test, so each test can prepare its files first and load the module second.

Timestamps with and without microseconds

An Apache error log line starts with a time in one of two shapes: with fractional seconds when the server was built with them, without them when it was not. The analyzer tries both formats and returns None for anything else, and the first test pins all three outcomes.

The module container/tools/analyze_logs/analyze_errors.py:

_TS_FORMATS = ("%a %b %d %H:%M:%S.%f %Y", "%a %b %d %H:%M:%S %Y") def parse_ts(ts: str) -> datetime | None: """Apache error-log time such as "Thu Oct 22 21:11:34.663857 2026"; None when unknown.""" for fmt in _TS_FORMATS: try: return datetime.strptime(ts.strip(), fmt) except ValueError: continue return None

The test container/tests/tools/test_analyze_errors.py, first test:

def test_timestamps_with_and_without_microseconds_are_read(): analyze = _module() assert analyze.parse_ts("Thu Oct 22 21:11:34.663857 2026") == datetime(2026, 10, 22, 21, 11, 34, 663857) assert analyze.parse_ts("Fri Oct 02 11:00:00 2026") == datetime(2026, 10, 2, 11, 0, 0) assert analyze.parse_ts("not a time") is None

A single format would pass the first assertion and fail the second, and a parser that only caught ValueError inside one try would raise on the third. The None is not a detail: the --since filter later keeps a record without a readable time instead of dropping it, so an unparsed line is visible in the report rather than lost.

A report with first and last seen

The second test runs the real main() over three log lines and reads what it printed. Two helpers build the input: one line of a fatal error, and a directory with an error log, an empty src folder and a botlist.csv.

def _fatal(ts: str, client: str, rel: str) -> str: return f'[{ts}] [error] [client {client}:1234] PHP Fatal error: Uncaught Error: Class "RelatedArticles" not found in {PREFIX}{rel}:12\n' def _logs(tmp_path: Path) -> Path: log = tmp_path / "error.log" log.write_text( _fatal("Thu Oct 01 10:00:00.123456 2026", "203.0.113.5", "old/index.php") + _fatal("Fri Oct 02 11:00:00 2026", "203.0.113.6", "old/index.php") + _fatal("Thu Oct 22 09:30:00.000001 2026", "203.0.113.7", "current/index.php"), encoding="utf-8", ) (tmp_path / "src").mkdir() (tmp_path / "botlist.csv").write_text("ip\n", encoding="utf-8") return log

The docroot prefix in PREFIX is the default one the module knows, so it can cut the path of the failing script out of the message. The botlist holds only its header: no client is filtered, and the three records reach the report. A missing src folder would make main() exit with 1, which is why the helper creates an empty one.

def test_report_shows_first_and_last_seen(tmp_path, capsys): analyze = _module() log = _logs(tmp_path) assert analyze.main(["--log-file", str(log), "--src", str(tmp_path / "src"), "--botlist", str(tmp_path / "botlist.csv")]) == 0 out = capsys.readouterr().out assert "[2x, 2 clients] old/index.php\n seen: first 2026-10-01 10:00, last 2026-10-02 11:00" in out assert "[1x, 1 clients] current/index.php\n seen: first 2026-10-22 09:30, last 2026-10-22 09:30" in out

capsys is the pytest fixture that captures everything written to standard output and standard error. readouterr() returns what was captured so far and resets the buffer, so the assertions see exactly the report of this call.

The two expected fragments are the whole grouping logic. Two lines about old/index.php with two different client addresses become one group of 2x and two clients; the third line about current/index.php is its own group. first and last are the smallest and the largest timestamp of the group, printed to the minute. The group with more hits comes first, which is why the older page is asserted first.

--since: keep what happened after a deploy

After a deploy, the interesting errors are the ones that still happen. The option takes a date, or a date with hours and minutes, and drops every record older than that moment. The test checks the count line, the kept group, the dropped group and the exit code for a value that is not a date.

def test_since_drops_errors_from_before_a_deploy(tmp_path, capsys): analyze = _module() log = _logs(tmp_path) assert analyze.main(["--log-file", str(log), "--src", str(tmp_path / "src"), "--botlist", str(tmp_path / "botlist.csv"), "--since", "2026-10-15"]) == 0 out = capsys.readouterr().out assert "since 2026-10-15 00:00: dropped 2 older records" in out assert "current/index.php" in out assert "old/index.php" not in out assert analyze.main(["--log-file", str(log), "--src", str(tmp_path / "src"), "--botlist", str(tmp_path / "botlist.csv"), "--since", "15.10.2026"]) == 2

Two October records are older than the fifteenth and one is not, so the note says dropped 2 older records. The two membership assertions are the ones that matter: the surviving path must be in the report and the dropped path must be gone, while old/index.php being absent alone would also be true for an empty report.

The last line is about the other side of the option. A value the analyzer cannot read, here with dots and the day first, raises ValueError inside parse_since(); main() catches it, prints the reason to standard error and returns 2. The test asserts the code, not the message, and does not need capsys for it.

--check-prod: one request per fatal page

The last test is the only one that would talk to the network, so it replaces the single function that does the request. The option asks prod for every page with a PHP fatal error, most hits first, and stops at a cap so a bad day does not turn into a hundred requests against the live site.

def test_check_prod_asks_prod_once_per_fatal_page_within_the_cap(tmp_path, monkeypatch): analyze = _module() asked: list[str] = [] monkeypatch.setattr(analyze, "check_dev_url", lambda url, timeout=10: asked.append(url) or "200") records = [analyze.parse_line(line) for line in _logs(tmp_path).read_text(encoding="utf-8").splitlines()] report = analyze.analyze(records, str(tmp_path / "src"), "https://dev.example.invalid", False, 0, 30, "https://prod.example.invalid", True, 1) assert asked == ["https://prod.example.invalid/old/"] assert "prod now: https://prod.example.invalid/old/ -> 200" in report assert "prod now: https://prod.example.invalid/current/" not in report

The lambda records the URL and returns the status string "200": asked.append(url) returns None, so or "200" supplies the value the analyzer expects. monkeypatch.setattr replaces the function on the loaded module, and the code under test finds it there, because check_prod_for_groups() looks the name up as a global of its own module at call time.

The call to analyze() passes its nine arguments by position: records, src, the dev base with check_dev=False, zero dev checks, a top of thirty, the prod base, check_prod=True and a cap of one. Two fatal groups would qualify, and exactly one URL is asked for, which is what asked == [...] proves.

The URL ends with /old/ and not with /old/index.php: a directory page is requested as a directory. The last assertion is the cap again from the report side: the second group has no prod now: line at all.

Run the file

Run the file from the container directory:

./tools/run-tests-venv.sh -- tests/tools/test_analyze_errors.py -q

Checked 5 packages in 3ms BASE_URL=http://127.0.0.1:8080 SITEMAP_FILE=src/sitemap.xml .... [100%] 4 passed in 1.20s

The first line is uv checking the packages of the venv, the second the two variables the script exports for the tests. The four dots are the four tests in file order: the timestamp parser, the report, --since and --check-prod. Nothing was fetched and nothing was written outside the temporary directory.

Article author: Arthur Isaev