Testing the error-log analyzer: dates, --since and --check-prod
Introduction
analyze_errors.py reads the Apache error logs of prod,
groups identical errors and answers four questions per group: is the
address a real page or a wrong one, is the failure reproducible from the
local src tree, when was it first and last seen, and
what does prod answer for that page right now.
Its test, container/tests/tools/test_analyze_errors.py,
has four tests. Each one takes a part of that pipeline: the timestamp
parser, the report lines with first and last seen, the
--since filter that drops everything older than a
deploy, and the --check-prod option with its cap.
There is no log directory, no network and no
PHP
: the log lines are written
by the test, and the client addresses are from the documentation range
203.0.113.0/24.
Tools used here: Python , pytest .
Load the module by path
The analyzer is a script in container/tools/analyze_logs, not an installed package, and it imports its neighbour log_paths.py. A plain import would fail twice: the folder is not on sys.path, and the module is not on any path at all. The test loads it from its file and adds the folder to the import path only while the module body runs.
The test file container/tests/tools/test_analyze_errors.py, header and loader:
from __future__ import annotations import importlib.util import sys from datetime import datetime from pathlib import Path ANALYZE_LOGS = Path(__file__).resolve().parents[2] / "tools" / "analyze_logs" PREFIX = "/home/a/andreyv/aredel.com/public_html/" def _module(): sys.path.insert(0, str(ANALYZE_LOGS)) try: spec = importlib.util.spec_from_file_location("aredel_analyze_errors", ANALYZE_LOGS / "analyze_errors.py") assert spec and spec.loader module = importlib.util.module_from_spec(spec) sys.modules[spec.name] = module # dataclasses look their module up here spec.loader.exec_module(module) return module finally: sys.path.remove(str(ANALYZE_LOGS))
The importlib recipe is the usual three calls: build a spec from the path, build a module object from the spec, run the file in that object. Two details belong to this module in particular:
- sys.path.insert(0, ...) is what lets from log_paths import prod_error_dir inside the analyzer resolve: the folder is on the path for the duration of exec_module() and removed in finally, so no other test inherits it;
- sys.modules[spec.name] = module registers the module before its body runs, because the two @dataclass classes in the file ask sys.modules for the module they were defined in. Without the line the import fails with an AttributeError about __module__.
The loader is called by every test, so each test can prepare its files first and load the module second.
Timestamps with and without microseconds
An Apache error log line starts with a time in one of two shapes: with fractional seconds when the server was built with them, without them when it was not. The analyzer tries both formats and returns None for anything else, and the first test pins all three outcomes.
The module container/tools/analyze_logs/analyze_errors.py:
_TS_FORMATS = ("%a %b %d %H:%M:%S.%f %Y", "%a %b %d %H:%M:%S %Y") def parse_ts(ts: str) -> datetime | None: """Apache error-log time such as "Thu Oct 22 21:11:34.663857 2026"; None when unknown.""" for fmt in _TS_FORMATS: try: return datetime.strptime(ts.strip(), fmt) except ValueError: continue return None
The test container/tests/tools/test_analyze_errors.py, first test:
def test_timestamps_with_and_without_microseconds_are_read(): analyze = _module() assert analyze.parse_ts("Thu Oct 22 21:11:34.663857 2026") == datetime(2026, 10, 22, 21, 11, 34, 663857) assert analyze.parse_ts("Fri Oct 02 11:00:00 2026") == datetime(2026, 10, 2, 11, 0, 0) assert analyze.parse_ts("not a time") is None
A single format would pass the first assertion and fail the second, and a parser that only caught ValueError inside one try would raise on the third. The None is not a detail: the --since filter later keeps a record without a readable time instead of dropping it, so an unparsed line is visible in the report rather than lost.
A report with first and last seen
The second test runs the real main() over three log lines and reads what it printed. Two helpers build the input: one line of a fatal error, and a directory with an error log, an empty src folder and a botlist.csv.
def _fatal(ts: str, client: str, rel: str) -> str: return f'[{ts}] [error] [client {client}:1234] PHP Fatal error: Uncaught Error: Class "RelatedArticles" not found in {PREFIX}{rel}:12\n' def _logs(tmp_path: Path) -> Path: log = tmp_path / "error.log" log.write_text( _fatal("Thu Oct 01 10:00:00.123456 2026", "203.0.113.5", "old/index.php") + _fatal("Fri Oct 02 11:00:00 2026", "203.0.113.6", "old/index.php") + _fatal("Thu Oct 22 09:30:00.000001 2026", "203.0.113.7", "current/index.php"), encoding="utf-8", ) (tmp_path / "src").mkdir() (tmp_path / "botlist.csv").write_text("ip\n", encoding="utf-8") return log
The docroot prefix in PREFIX is the default one the module knows, so it can cut the path of the failing script out of the message. The botlist holds only its header: no client is filtered, and the three records reach the report. A missing src folder would make main() exit with 1, which is why the helper creates an empty one.
def test_report_shows_first_and_last_seen(tmp_path, capsys): analyze = _module() log = _logs(tmp_path) assert analyze.main(["--log-file", str(log), "--src", str(tmp_path / "src"), "--botlist", str(tmp_path / "botlist.csv")]) == 0 out = capsys.readouterr().out assert "[2x, 2 clients] old/index.php\n seen: first 2026-10-01 10:00, last 2026-10-02 11:00" in out assert "[1x, 1 clients] current/index.php\n seen: first 2026-10-22 09:30, last 2026-10-22 09:30" in out
capsys is the
pytest fixture
that captures
everything written to standard output and standard error.
readouterr() returns what was captured so far and
resets the buffer, so the assertions see exactly the report of this call.
The two expected fragments are the whole grouping logic. Two lines about
old/index.php with two different client addresses
become one group of 2x and two clients; the third
line about current/index.php is its own group.
first and last are the
smallest and the largest timestamp of the group, printed to the minute.
The group with more hits comes first, which is why the older page is
asserted first.
--since: keep what happened after a deploy
After a deploy, the interesting errors are the ones that still happen. The option takes a date, or a date with hours and minutes, and drops every record older than that moment. The test checks the count line, the kept group, the dropped group and the exit code for a value that is not a date.
def test_since_drops_errors_from_before_a_deploy(tmp_path, capsys): analyze = _module() log = _logs(tmp_path) assert analyze.main(["--log-file", str(log), "--src", str(tmp_path / "src"), "--botlist", str(tmp_path / "botlist.csv"), "--since", "2026-10-15"]) == 0 out = capsys.readouterr().out assert "since 2026-10-15 00:00: dropped 2 older records" in out assert "current/index.php" in out assert "old/index.php" not in out assert analyze.main(["--log-file", str(log), "--src", str(tmp_path / "src"), "--botlist", str(tmp_path / "botlist.csv"), "--since", "15.10.2026"]) == 2
Two October records are older than the fifteenth and one is not, so the
note says dropped 2 older records. The two
membership assertions are the ones that matter: the surviving path must be
in the report and the dropped path must be gone, while
old/index.php being absent alone would also be true
for an empty report.
The last line is about the other side of the option. A value the analyzer
cannot read, here with dots and the day first, raises
ValueError inside
parse_since(); main()
catches it, prints the reason to standard error and returns 2. The test
asserts the code, not the message, and does not need
capsys for it.
--check-prod: one request per fatal page
The last test is the only one that would talk to the network, so it replaces the single function that does the request. The option asks prod for every page with a PHP fatal error, most hits first, and stops at a cap so a bad day does not turn into a hundred requests against the live site.
def test_check_prod_asks_prod_once_per_fatal_page_within_the_cap(tmp_path, monkeypatch): analyze = _module() asked: list[str] = [] monkeypatch.setattr(analyze, "check_dev_url", lambda url, timeout=10: asked.append(url) or "200") records = [analyze.parse_line(line) for line in _logs(tmp_path).read_text(encoding="utf-8").splitlines()] report = analyze.analyze(records, str(tmp_path / "src"), "https://dev.example.invalid", False, 0, 30, "https://prod.example.invalid", True, 1) assert asked == ["https://prod.example.invalid/old/"] assert "prod now: https://prod.example.invalid/old/ -> 200" in report assert "prod now: https://prod.example.invalid/current/" not in report
The lambda records the URL and returns the status string
"200":
asked.append(url) returns
None, so
or "200" supplies the value the analyzer
expects. monkeypatch.setattr replaces the function
on the loaded module, and the code under test finds it there, because
check_prod_for_groups() looks the name up as a
global of its own module at call time.
The call to analyze() passes its nine arguments by
position: records, src, the dev base with
check_dev=False, zero dev checks, a top of thirty,
the prod base, check_prod=True and a cap of one.
Two fatal groups would qualify, and exactly one URL is asked for, which is
what asked == [...] proves.
The URL ends with /old/ and not with
/old/index.php: a directory page is requested as a
directory. The last assertion is the cap again from the report side: the
second group has no prod now: line at all.
Run the file
Run the file from the container directory:
./tools/run-tests-venv.sh -- tests/tools/test_analyze_errors.py -q
Checked 5 packages in 3ms BASE_URL=http://127.0.0.1:8080 SITEMAP_FILE=src/sitemap.xml .... [100%] 4 passed in 1.20s
The first line is uv checking the packages of the venv, the second the two variables the script exports for the tests. The four dots are the four tests in file order: the timestamp parser, the report, --since and --check-prod. Nothing was fetched and nothing was written outside the temporary directory.
Article author: Arthur Isaev