Testing the bot signatures of check_bots.py

Introduction

check_bots.py rebuilds botlist.csv from the prod access logs. Most of its rules need volume: a known bot user agent on most of the hits, ten or more 404 answers with a 404 share of at least half, ten or more probe paths, a single day with a hundred and fifty hits.

One scraper defeats all of them. It asks for about one page per address, with a different 2017-2019 browser user agent each time, so no counter reaches a threshold. What it cannot hide is what it asks for: a card image name glued onto a page path, or a Referer holding two URLs. One such request is enough to list the address, and that rule is what container/tests/tools/test_check_bots.py tests.

The file has four tests and seventeen items: two parametrized tables with eight and seven rows, one end-to-end run over a dummy access log, and one test of the CSV write. The pipeline calls the script from refresh-metrika.sh and analyze_errors.py reads the file it writes.

Tools used here: Python , pytest .

Two signatures in the module

Both signatures are one regular expression and one small function. The first looks at the requested path, the second at the Referer header.

The module container/tools/analyze_logs/check_bots.py:

CARD_PATH_RE = re.compile(r"/it/[\w-]+_card/?$") FORGED_REFERER_RE = re.compile(r"^https?://\S*,\s+https?://", re.IGNORECASE) def is_card_path(target: str) -> bool: """True for a card image name glued onto a page path (/x/it/gulp_card). Pages only link the card images as /img/cards/it/<name>_card.<ext>; the extensionless it/<name>_card under a page comes from a scraper that took the names from old andreyolegovich.ru pages. """ return bool(CARD_PATH_RE.search(urlsplit(target).path)) def is_forged_referer(referer: str) -> bool: """True for a Referer holding two comma-separated URLs; browsers send one.""" return bool(FORGED_REFERER_RE.match(referer))

The card expression asks for the literal folder /it/, then a name of word characters and hyphens, then _card and an optional slash at the very end of the path. The search runs on urlsplit(target).path, so a query string is out of the way before the match, and search() lets any page prefix come before /it/.

The Referer expression is anchored at the start: one URL, a comma with at least one space after it, and a second http or https URL. A browser sends one URL there; two are what the scraper forged from the old site.

Card names glued onto page paths

The first table feeds eight paths to is_card_path() and states the expected answer for each. The parametrize marker turns the table into eight test items, and pytest prints the row values in the id of each one.

@pytest.mark.parametrize( ("target", "expected"), [ ("/ru/code/python/packages/layout/it/postgresql_card", True), ("/ru/qa/it/gulp_card", True), ("/testing/robot/it/ms_powerpoint_card?x=1", True), ("/sport/injuries/meniscus/it/testing_edu_card/", True), ("/img/cards/it/postgresql_card.png", False), ("/img/cards//it/postgresql_card.webp", False), ("/ru/i/postgres/", False), ("/it/", False), ], ) def test_is_card_path(target: str, expected: bool) -> None: assert bots.is_card_path(target) is expected

The four true rows are four shapes of the same request: a card name under a deep page path, under a short one, with a query string, and with a trailing slash. The four false rows guard the other side:

is expected is deliberate instead of == expected: the function must return a real boolean, not a match object or a non-empty string that merely looks true in an if.

A Referer with two URLs

@pytest.mark.parametrize( ("referer", "expected"), [ ("https://google.com, https://www.urn.su/testing/robot/?utm_source=andreyolegovich.ru&utm_campaign=rblock", True), ("https://google.com, https://andreyolegovich.ru/windows/network/telnet/", True), ("-", False), ("", False), ("https://www.urn.su/testing/jira/", False), ("https://example.com/list?ids=1,2,3", False), ("https://example.com/?next=a,https://example.org/", False), ], ) def test_is_forged_referer(referer: str, expected: bool) -> None: assert bots.is_forged_referer(referer) is expected

The two true rows are the forged header as it appears in the logs: the first URL, a comma, a space, the second URL. The five false rows are the near misses:

One request lists an IP

The two tables test the functions. The third test runs main() over a real access log file and reads the CSV it writes. One helper builds one line in the format of the access logs of the server.

The test file container/tests/tools/test_check_bots.py, header and line builder:

from __future__ import annotations import csv import importlib.util import sys from pathlib import Path import pytest ANALYZE_LOGS = Path(__file__).resolve().parents[2] / "tools" / "analyze_logs" BROWSER_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2; rv:52.2.0) Gecko/20100101 Firefox/52.2.0" def _module(): sys.path.insert(0, str(ANALYZE_LOGS)) try: spec = importlib.util.spec_from_file_location("aredel_check_bots", ANALYZE_LOGS / "check_bots.py") assert spec and spec.loader module = importlib.util.module_from_spec(spec) sys.modules[spec.name] = module # dataclasses look their module up here spec.loader.exec_module(module) return module finally: sys.path.remove(str(ANALYZE_LOGS)) bots = _module() def _line(ip: str, target: str, status: int = 404, referer: str = "-") -> str: return f'aredel.com {ip} - - [30/Oct/2026:23:47:27 +0300] "GET {target} HTTP/1.1" {status} 5234 "{referer}" "{BROWSER_UA}" 26655 23709:3387\n'

The module is loaded once, at import of the test file: the tables only call two pure functions, and the end-to-end test passes its own directories to main(), so no test changes module state. The loader is the same three-call recipe as in the analyzer test, with the folder on sys.path while the body runs and the module registered in sys.modules for its dataclasses.

def test_one_signature_request_lists_the_ip(tmp_path: Path) -> None: logs = tmp_path / "access" logs.mkdir() (logs / "aredel.com.access.log").write_text( _line("203.0.113.10", "/ru/code/python/packages/layout/it/postgresql_card") + _line("203.0.113.11", "/ru/qa/robot/", 200, "https://google.com, https://www.urn.su/testing/robot/") + _line("203.0.113.12", "/ru/code/python/class/variables.php") + _line("203.0.113.13", "/img/cards/it/postgresql_card.png", 200, "https://aredel.com/ru/i/postgres/") + _line("198.51.100.20", "/ru/git"), encoding="utf-8", ) output = tmp_path / "botlist.csv" assert bots.main(["--log-dir", str(logs), "--output", str(output)]) == 0 with open(output, encoding="utf-8", newline="") as handle: listed = {row["ip"] for row in csv.DictReader(handle)} assert listed == {"203.0.113.10", "203.0.113.11"}

Five requests from five addresses, and only two of them are listed. Read the log line by line:

The assertion is a set, not a list, so the order of the rows in the CSV does not matter. newline="" on open() is required by the csv module: without it a quoted field that contains a line break would be read back with doubled characters on some platforms.

A failed write keeps the old botlist

The botlist is rebuilt by every cron run and read at the same time by the metrika builder and the error analyzer. A half-written file would make both read a truncated list, so the module writes to a temporary name and renames it over the real one.

def write_botlist(rows: list[dict[str, Any]], output: str) -> None: """Write rows to botlist.csv with the canonical columns. Written to <output>.tmp.<pid> and renamed over output, so the readers (build_metrika_db.py, analyze_errors.py) never see a cut-off list. """ tmp_path = f"{output}.tmp.{os.getpid()}" try: with open(tmp_path, "w", encoding="utf-8", newline="") as handle: writer = csv.DictWriter( handle, fieldnames=["ip", "hits_404", "distinct_paths", "first_seen", "last_seen", "top_user_agent", "sample_paths"], ) writer.writeheader() writer.writerows(rows) os.replace(tmp_path, output) except BaseException: if os.path.exists(tmp_path): os.remove(tmp_path) raise

The last test breaks the write on purpose: a row carries a key that is not one of the columns, and csv.DictWriter refuses extra keys by default.

def test_failed_write_keeps_the_old_botlist(tmp_path: Path) -> None: output = tmp_path / "botlist.csv" output.write_text("ip\n203.0.113.10\n", encoding="utf-8") with pytest.raises(ValueError): bots.write_botlist([{"ip": "203.0.113.11", "not_a_column": 1}], str(output)) assert output.read_text(encoding="utf-8") == "ip\n203.0.113.10\n" assert [path.name for path in tmp_path.iterdir()] == ["botlist.csv"]

The writer raises ValueError while it writes the rows, after the header has already reached the temporary file. The handler removes that file and re-raises, so the exception is what the test expects, and os.replace is never reached.

Both assertions matter. The content check proves the old list still holds the address it had; the directory check proves no botlist.csv.tmp.<pid> was left behind for the next run to trip over. A test that only expected the exception would pass for a module that had truncated the real file before failing.

Run the file

Run the file from the container directory:

./tools/run-tests-venv.sh -- tests/tools/test_check_bots.py -q

Checked 5 packages in 3ms BASE_URL=http://127.0.0.1:8080 SITEMAP_FILE=src/sitemap.xml ................. [100%] 17 passed in 1.11s

The first line is uv checking the packages of the venv, the second the two variables the script exports for the tests. Seventeen dots: eight rows of the card table, seven of the Referer table, the end-to-end run and the write. To see which row is which, add -v; pytest then prints one line per item with the parameter values in the id, for example test_is_card_path[/ru/qa/it/gulp_card-True].

Article author: Arthur Isaev

Banner image