Testing the bot signatures of check_bots.py
Introduction
check_bots.py rebuilds botlist.csv
from the prod access logs. Most of its rules need volume: a known bot user
agent on most of the hits, ten or more 404 answers with a 404 share of at
least half, ten or more probe paths, a single day with a hundred and fifty
hits.
One scraper defeats all of them. It asks for about one page per address,
with a different 2017-2019 browser user agent each time, so no counter
reaches a threshold. What it cannot hide is what it asks for: a card image
name glued onto a page path, or a Referer holding two URLs. One such
request is enough to list the address, and that rule is what
container/tests/tools/test_check_bots.py tests.
The file has four tests and seventeen items: two parametrized tables with
eight and seven rows, one end-to-end run over a dummy access log, and one
test of the CSV write. The pipeline calls the script from
refresh-metrika.sh
and
analyze_errors.py
reads the file it writes.
Tools used here: Python , pytest .
Two signatures in the module
Both signatures are one regular expression and one small function. The first looks at the requested path, the second at the Referer header.
The module container/tools/analyze_logs/check_bots.py:
CARD_PATH_RE = re.compile(r"/it/[\w-]+_card/?$") FORGED_REFERER_RE = re.compile(r"^https?://\S*,\s+https?://", re.IGNORECASE) def is_card_path(target: str) -> bool: """True for a card image name glued onto a page path (/x/it/gulp_card). Pages only link the card images as /img/cards/it/<name>_card.<ext>; the extensionless it/<name>_card under a page comes from a scraper that took the names from old andreyolegovich.ru pages. """ return bool(CARD_PATH_RE.search(urlsplit(target).path)) def is_forged_referer(referer: str) -> bool: """True for a Referer holding two comma-separated URLs; browsers send one.""" return bool(FORGED_REFERER_RE.match(referer))
The card expression asks for the literal folder
/it/, then a name of word characters and hyphens, then
_card and an optional slash at the very end of the
path. The search runs on urlsplit(target).path, so
a query string is out of the way before the match, and
search() lets any page prefix come before
/it/.
The Referer expression is anchored at the start: one URL, a comma with at
least one space after it, and a second http or
https URL. A browser sends one URL there; two
are what the scraper forged from the old site.
Card names glued onto page paths
The first table feeds eight paths to is_card_path() and states the expected answer for each. The parametrize marker turns the table into eight test items, and pytest prints the row values in the id of each one.
@pytest.mark.parametrize( ("target", "expected"), [ ("/ru/code/python/packages/layout/it/postgresql_card", True), ("/ru/qa/it/gulp_card", True), ("/testing/robot/it/ms_powerpoint_card?x=1", True), ("/sport/injuries/meniscus/it/testing_edu_card/", True), ("/img/cards/it/postgresql_card.png", False), ("/img/cards//it/postgresql_card.webp", False), ("/ru/i/postgres/", False), ("/it/", False), ], ) def test_is_card_path(target: str, expected: bool) -> None: assert bots.is_card_path(target) is expected
The four true rows are four shapes of the same request: a card name under a deep page path, under a short one, with a query string, and with a trailing slash. The four false rows guard the other side:
- /img/cards/it/postgresql_card.png is the real image URL, with an extension. Pages link that one, so it must never count as a signature, and the same holds for the .webp row with its double slash;
- /ru/i/postgres/ is a normal page: no /it/ folder, no card name;
- /it/ is the folder alone, without a name.
is expected is deliberate instead of == expected: the function must return a real boolean, not a match object or a non-empty string that merely looks true in an if.
A Referer with two URLs
@pytest.mark.parametrize( ("referer", "expected"), [ ("https://google.com, https://www.urn.su/testing/robot/?utm_source=andreyolegovich.ru&utm_campaign=rblock", True), ("https://google.com, https://andreyolegovich.ru/windows/network/telnet/", True), ("-", False), ("", False), ("https://www.urn.su/testing/jira/", False), ("https://example.com/list?ids=1,2,3", False), ("https://example.com/?next=a,https://example.org/", False), ], ) def test_is_forged_referer(referer: str, expected: bool) -> None: assert bots.is_forged_referer(referer) is expected
The two true rows are the forged header as it appears in the logs: the first URL, a comma, a space, the second URL. The five false rows are the near misses:
- - is what the server writes when a request has no Referer, and "" is an empty header; both must stay false, otherwise every direct request would list its address;
- one ordinary URL stays false;
- https://example.com/list?ids=1,2,3 has commas, but they separate values inside a query string and are not followed by a space and a scheme;
- https://example.com/?next=a,https://example.org/ is the closest call: the comma is followed by a real URL, but with no space between them. The regular expression needs the whitespace, and a browser that puts a URL into a parameter does not insert one.
One request lists an IP
The two tables test the functions. The third test runs main() over a real access log file and reads the CSV it writes. One helper builds one line in the format of the access logs of the server.
The test file container/tests/tools/test_check_bots.py, header and line builder:
from __future__ import annotations import csv import importlib.util import sys from pathlib import Path import pytest ANALYZE_LOGS = Path(__file__).resolve().parents[2] / "tools" / "analyze_logs" BROWSER_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2; rv:52.2.0) Gecko/20100101 Firefox/52.2.0" def _module(): sys.path.insert(0, str(ANALYZE_LOGS)) try: spec = importlib.util.spec_from_file_location("aredel_check_bots", ANALYZE_LOGS / "check_bots.py") assert spec and spec.loader module = importlib.util.module_from_spec(spec) sys.modules[spec.name] = module # dataclasses look their module up here spec.loader.exec_module(module) return module finally: sys.path.remove(str(ANALYZE_LOGS)) bots = _module() def _line(ip: str, target: str, status: int = 404, referer: str = "-") -> str: return f'aredel.com {ip} - - [30/Oct/2026:23:47:27 +0300] "GET {target} HTTP/1.1" {status} 5234 "{referer}" "{BROWSER_UA}" 26655 23709:3387\n'
The module is loaded once, at import of the test file: the tables only call two pure functions, and the end-to-end test passes its own directories to main(), so no test changes module state. The loader is the same three-call recipe as in the analyzer test, with the folder on sys.path while the body runs and the module registered in sys.modules for its dataclasses.
def test_one_signature_request_lists_the_ip(tmp_path: Path) -> None: logs = tmp_path / "access" logs.mkdir() (logs / "aredel.com.access.log").write_text( _line("203.0.113.10", "/ru/code/python/packages/layout/it/postgresql_card") + _line("203.0.113.11", "/ru/qa/robot/", 200, "https://google.com, https://www.urn.su/testing/robot/") + _line("203.0.113.12", "/ru/code/python/class/variables.php") + _line("203.0.113.13", "/img/cards/it/postgresql_card.png", 200, "https://aredel.com/ru/i/postgres/") + _line("198.51.100.20", "/ru/git"), encoding="utf-8", ) output = tmp_path / "botlist.csv" assert bots.main(["--log-dir", str(logs), "--output", str(output)]) == 0 with open(output, encoding="utf-8", newline="") as handle: listed = {row["ip"] for row in csv.DictReader(handle)} assert listed == {"203.0.113.10", "203.0.113.11"}
Five requests from five addresses, and only two of them are listed. Read the log line by line:
- 203.0.113.10 asks for one card path and nothing else. Volume rules see one hit and one 404; the signature rule sees the path and lists the address;
- 203.0.113.11 asks for a normal page and answers 200, but sends the forged two-URL Referer. The signature does not care about the status, so the address is listed too;
- 203.0.113.12 asks once for a missing page: a 404 without a signature, far below the ten-hit threshold;
- 203.0.113.13 asks for the real card image, with the extension, and sends one ordinary Referer. This is the control for the false rows of the card table : a browser that loads a card image must never be listed;
- 198.51.100.20 is the second plain 404, from the other documentation range.
The assertion is a set, not a list, so the order of the rows in the CSV does not matter. newline="" on open() is required by the csv module: without it a quoted field that contains a line break would be read back with doubled characters on some platforms.
A failed write keeps the old botlist
The botlist is rebuilt by every cron run and read at the same time by the metrika builder and the error analyzer. A half-written file would make both read a truncated list, so the module writes to a temporary name and renames it over the real one.
def write_botlist(rows: list[dict[str, Any]], output: str) -> None: """Write rows to botlist.csv with the canonical columns. Written to <output>.tmp.<pid> and renamed over output, so the readers (build_metrika_db.py, analyze_errors.py) never see a cut-off list. """ tmp_path = f"{output}.tmp.{os.getpid()}" try: with open(tmp_path, "w", encoding="utf-8", newline="") as handle: writer = csv.DictWriter( handle, fieldnames=["ip", "hits_404", "distinct_paths", "first_seen", "last_seen", "top_user_agent", "sample_paths"], ) writer.writeheader() writer.writerows(rows) os.replace(tmp_path, output) except BaseException: if os.path.exists(tmp_path): os.remove(tmp_path) raise
The last test breaks the write on purpose: a row carries a key that is not one of the columns, and csv.DictWriter refuses extra keys by default.
def test_failed_write_keeps_the_old_botlist(tmp_path: Path) -> None: output = tmp_path / "botlist.csv" output.write_text("ip\n203.0.113.10\n", encoding="utf-8") with pytest.raises(ValueError): bots.write_botlist([{"ip": "203.0.113.11", "not_a_column": 1}], str(output)) assert output.read_text(encoding="utf-8") == "ip\n203.0.113.10\n" assert [path.name for path in tmp_path.iterdir()] == ["botlist.csv"]
The writer raises ValueError while it writes the
rows, after the header has already reached the temporary file. The handler
removes that file and re-raises, so the exception is what the test expects,
and os.replace is never reached.
Both assertions matter. The content check proves the old list still holds
the address it had; the directory check proves no
botlist.csv.tmp.<pid> was left behind for the
next run to trip over. A test that only expected the exception would pass
for a module that had truncated the real file before failing.
Run the file
Run the file from the container directory:
./tools/run-tests-venv.sh -- tests/tools/test_check_bots.py -q
Checked 5 packages in 3ms BASE_URL=http://127.0.0.1:8080 SITEMAP_FILE=src/sitemap.xml ................. [100%] 17 passed in 1.11s
The first line is uv checking the packages of the venv, the second the two variables the script exports for the tests. Seventeen dots: eight rows of the card table, seven of the Referer table, the end-to-end run and the write. To see which row is which, add -v; pytest then prints one line per item with the parameter values in the id, for example test_is_card_path[/ru/qa/it/gulp_card-True].
Article author: Arthur Isaev