Parametrization in PyTest

Introduction

Instead of running the same test with different input data, it is more convenient to make one test that takes several sets of arguments.

Creating such generalizing tests is what is usually called parametrization .

This approach is then easier to scale. Also, when there are steps that are expensive to compute, repeating them will hurt performance.

For example, it is much more efficient to connect to the database at the beginning of the test and close the connection at the end than to open and close a connection for every data set.

Official documentation

The counterpart of this article for Robot Framework can be studied here .

The solution to the module visibility problem is here .

For all the examples to run, it is desirable to do something like

python -m venv venv
venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install pytest requests xmlschema

A small demonstration of how the @pytest.mark.parametrize decorator works; it is what parametrization in PyTest is done with

import pytest @pytest.mark.parametrize("test_input, expected_result", [ ("99 > 99", False), ("100 > 99", True), ("2*2", 5), ("1+2", 3) ]) def test_compare(test_input, expected_result): assert eval(test_input) == expected_result

======================== test session starts ========================== collected 4 items tests/test_app.py::test_compare[99 > 99-False] PASSED [ 25%] tests/test_app.py::test_compare[100 > 99-True] PASSED [ 50%] tests/test_app.py::test_compare[2*2-5] FAILED [ 75%] tests/test_app.py::test_compare[1+2-3] PASSED [100%] ============================== FAILURES ============================== ________________________ test_compare[2*2-5] _________________________ …

In this example we checked expressions that were evaluated with eval right in the test.

This example serves to demonstrate pytest.mark.parametrize; you can study a way of working with such strings without using eval() here

In the next example we will pass the numbers separately.

The expected result is a check that the first number is strictly greater than the second.

import pytest @pytest.mark.parametrize( "entry1, entry2, expected_result", [ (99, 100, False), (100, 99, True), (99, 99, False), (-2, 1, False), ], ) def test_compare(entry1, entry2, expected_result): assert (entry1 > entry2) is expected_result

… collected 4 items two.py::test_compare[99-100-False] PASSED [ 25%] two.py::test_compare[100-99-True] PASSED [ 50%] two.py::test_compare[99-99-False] PASSED [ 75%] two.py::test_compare[-2-1-False] PASSED [100%] ========================== 4 passed in 0.04s ==========================

In reality you usually have to test the work of some external functions. We will go through several examples in the following chapters.

Checking a list of unknown length

The get_digits() function from the list_of_digits.py module must return an arbitrary set of digits.

Repetitions are allowed.

The digits are 0, 1, 2… 9.

The function is written with a bug, but it is not easy to catch with a single test.

I want to draw special attention to the fact that we are parametrizing a check of a list of a length unknown in advance.

With PyTest you can easily get a number of tests that is not determined in advance. And even a random one, if needed. As in our example.

Project structure

list_of_digits/ ├── list_of_digits.py └── tests └── pytest └── test_list_of_digits.py

# list_of_digits.py import random def get_digits() -> list: n = random.randint(1, 10) result = [] for i in range(0, n): result.append(random.randint(4, 10)) return result

To make the test more interesting, the function returns lists of pseudorandom length. The longer the list, the better the chances of catching the bug.

# test_list_of_digits.py import pytest import random from list_of_digits import get_digits digits = get_digits() @pytest.mark.parametrize("digit", digits) def test_if_digit(digit): assert 0 <= digit < 10

Let us run the tests from the list_of_digits directory several times until we catch the error.

python -m pytest -v --no-header .\tests\pytest\test_list_of_digits.py

=============== test session starts =============== collected 2 items tests/pytest/test_list_of_digits.py::test_if_digit[8] PASSED [ 50%] tests/pytest/test_list_of_digits.py::test_if_digit[7] PASSED [100%] ================ 2 passed in 0.01s ================

=============== test session starts =============== collected 10 items tests/pytest/test_list_of_digits.py::test_if_digit[9_0] PASSED [ 10%] tests/pytest/test_list_of_digits.py::test_if_digit[4] PASSED [ 20%] tests/pytest/test_list_of_digits.py::test_if_digit[7] PASSED [ 30%] tests/pytest/test_list_of_digits.py::test_if_digit[8_0] PASSED [ 40%] tests/pytest/test_list_of_digits.py::test_if_digit[9_1] PASSED [ 50%] tests/pytest/test_list_of_digits.py::test_if_digit[8_1] PASSED [ 60%] tests/pytest/test_list_of_digits.py::test_if_digit[6_0] PASSED [ 70%] tests/pytest/test_list_of_digits.py::test_if_digit[8_2] PASSED [ 80%] tests/pytest/test_list_of_digits.py::test_if_digit[10] FAILED [ 90%] tests/pytest/test_list_of_digits.py::test_if_digit[6_1] PASSED [100%] ===================== FAILURES =====================

In the first test run a list of two elements was created. Checking it did not reveal any errors.

Thanks to @pytest.mark.parametrize each check was executed as a separate test. Therefore we see two PASSED results

In the second test run the list already had ten elements. We checked every element of the list.

The ninth element turned out to be 10, which is not a digit. Therefore the test failed.

This test shows well what PyTest does with the arguments passed to the decorator

@pytest.mark.parametrize("digit", digits)

The first argument, although passed as a string, becomes a variable of the same name responsible for iteration over the second argument.

I deliberately made an example with the simplest iteration of an ordinary variable over a list.

Without PyTest we could write the following check:

for digit in digits: assert 0 <= digit <= 9, f"{digit} is not a digit"

But it would fail at the first error, while with PyTest we see all the failures.

Three-digit numbers

Let us consider an example with slightly different input data. The project structure remains the same.

filter_three/ ├── three_digit.py └── tests └── pytest └── test_filter_three.py

The filter_three() function from the three_digit.py module returns True if the number has three digits.

def filter_three(x: int) -> bool: return x in range(100, 1000)

In the previous example we always expected the same result: every element had to be a digit.

Now, when testing the filter_three() function, we will pass both three-digit and non-three-digit numbers to it.

With each such test the expected result will also have to be passed.

Our test data can be represented as a list of tuples like this:

[(Number, Expected result), (Number, Expected result), (Number, Expected result)]

The key difference from the previous test is that here we have a fixed set of tests. In this example there are six of them.

Now, when we iterate over the test data, we will need two variables to unpack the tuples; let us call them test_input and expected_result

import pytest from three_digit import filter_three @pytest.mark.parametrize("test_input, expected_result", [ (8, False), (99, False), (100, True), (101, True), (999, True), (1000, False) ]) def test_compare(test_input, expected_result): res = filter_three(test_input) assert res == expected_result

python -m pytest -v --no-header .\tests\pytest\test_filter_three.py

======================== test session starts ========================== collected 6 items tests/pytest/test_filter_three.py::test_compare[9-False] PASSED [ 16%] tests/pytest/test_filter_three.py::test_compare[99-False] PASSED [ 33%] tests/pytest/test_filter_three.py::test_compare[100-True] PASSED [ 50%] tests/pytest/test_filter_three.py::test_compare[101-True] PASSED [ 66%] tests/pytest/test_filter_three.py::test_compare[999-True] PASSED [ 83%] tests/pytest/test_filter_three.py::test_compare[1000-False] PASSED [100%] ========================= 6 passed in 0.02s ===========================

Each test can be given its own name with id

import pytest from three_digit import filter_three @pytest.mark.parametrize("test_input, expected_result", [ pytest.param(8, False, id="Digit"), pytest.param(99, False, id="Two Digit Number"), pytest.param(100, True, id="Lower Border"), pytest.param(101, True, id="Three Digit Number"), pytest.param(999, True, id="Upper Border"), pytest.param(1000, False, id="Four Digit Number") ]) def test_compare(test_input, expected_result): res = filter_three(test_input) assert res == expected_result

======================== test session starts ========================== collected 6 items tests/pytest/test_filter_three.py::test_compare[Digit] PASSED [ 16%] tests/pytest/test_filter_three.py::test_compare[Two Digit Number] PASSED [ 33%] tests/pytest/test_filter_three.py::test_compare[Lower Border] PASSED [ 50%] tests/pytest/test_filter_three.py::test_compare[Three Digit Number] PASSED [ 66%] tests/pytest/test_filter_three.py::test_compare[Upper Border] PASSED [ 83%] tests/pytest/test_filter_three.py::test_compare[Four Digit Number] PASSED [100%] ========================= 6 passed in 0.02s ===========================

Multiplication of two numbers

Let us start with parametrizing a simple multiplication, the example we started to go through here .

Project structure:

parametrize/ ├── prod │ ├── prod.py │ └── tests │ └── pytest │ └── test_prod.py └── venv

# prod.py def prod(a, b): return a * b

We will run the tests from the prod directory with the command

python -m pytest -v --no-header .\tests\pytest\

Here the test data will be even more complex. We pass a list of tuples. Each tuple contains two elements. The first is a tuple of the two numbers to multiply. The second is the expected result as an int.

[((Multiplier, Multiplier), Expected result), ((Multiplier, Multiplier), Expected result), ((Multiplier, Multiplier), Expected result)]

# test_prod.py import pytest from prod import prod @pytest.mark.parametrize( "args, expected_result", [ ((0, 0), 0), ((3, 0), 0), ((-11, -12), 132), ]) def test_basic_param_prod(args, expected_result): res = prod(*args) assert res == expected_result

To avoid building up a complex unpacking, the elements of the tuple with the multipliers are iterated with *args

====================== test session starts ====================== collected 3 items tests/pytest/test_prod_no_id.py::test_basic_param_prod[args0-0] PASSED [ 33%] tests/pytest/test_prod_no_id.py::test_basic_param_prod[args1-0] PASSED [ 66%] tests/pytest/test_prod_no_id.py::test_basic_param_prod[args2-132] PASSED [100%] ==================== 3 passed in 0.01s =========================

For a clearer demonstration of the results, each test can be given a name with pytest.param id=

# test_prod.py import pytest from prod import prod @pytest.mark.parametrize( "args, expected_result", [ pytest.param((0, 0), 0, id="zero - zero"), pytest.param((7, -8), -56, id="positive - negative"), pytest.param((13.0, 14), 182, id="float positive - positive"), ]) def test_param_prod(args, expected_result): res = prod(*args) assert res == expected_result

====================== test session starts ====================== collected 3 items tests/pytest/test_prod.py::test_basic_param_prod[zero - zero] PASSED [ 33%] tests/pytest/test_prod.py::test_basic_param_prod[positive - negative] PASSED [ 66%] tests/pytest/test_prod.py::test_basic_param_prod[float positive - positive] PASSED [100%] ==================== 3 passed in 0.01s =========================

Quadratic equation

parametrize/ ├── quad │ ├── quadratic.py │ └── tests │ └── test_quadratic.py └── venv

We will run the tests from the quad directory

The simplest version of a parametrized test for solving a quadratic equation will look like this:

# Parametrization @pytest.mark.parametrize("args, expected_result",[ ((1, -3, -4), (4, -1)), ((0, 0, 0), (None, None)) ])

It is more convenient to give each test a name with pytest.param id=

Let us consider the full test code:

# test_quadratic.py import pytest from quadratic import quadratic_solve, TYPE_ERROR_TEXT def test_raises_type_error(): with pytest.raises(TypeError) as exc_info: quadratic_solve(1, "", 2) assert str(exc_info.value) == TYPE_ERROR_TEXT def test_result_is_tuple(): res = quadratic_solve(0, 0, 0) assert isinstance(res, tuple) @pytest.mark.parametrize("args, expected_result", [ pytest.param((1, -3, -4), (4, -1), id="two roots",), pytest.param((1, -2, 1), (1.0, None), id="single root",), pytest.param((0, 0, 0), (None, None), id="no roots",) ]) def test_solution(args, expected_result): res = quadratic_solve(*args) assert res == expected_result

python -m pytest --no-header -v tests/test_quadratic.py

============================ test session starts =============================== collected 5 items tests/test_quadratic.py::test_raises_type_error PASSED [ 20%] tests/test_quadratic.py::test_result_is_tuple PASSED [ 40%] tests/test_quadratic.py::test_solution[two roots] PASSED [ 60%] tests/test_quadratic.py::test_solution[single root] PASSED [ 80%] tests/test_quadratic.py::test_solution[no roots] PASSED [100%] ============================== 5 passed in 0.01s ==============================

As you can see, with the -v option all three tests [two roots], [single root], [no roots] pass successfully.

Site check

A practical example of using parametrization to check the responses of a site's pages.

The task is to check whether all pages of the heihei.ru site respond with HTTP code 200.

Project structure:

from_site_map/ ├── app │ ├── site_map.py │ └── tests │ └── pytest │ └── test_by_site_map.py └── venv

My hosting provider Beget.com creates a sitemap for free. I download it into the same directory where site_map.py is located.

from_site_map/ ├── app │ ├── site_map.py │ ├── sitemap.xml │ └── tests │ └── pytest │ └── test_by_site_map.py └── venv

The sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://heihei.ru/Spain/cities/malaga/</loc> <priority>0.8</priority> <changefreq>daily</changefreq> </url> <url> <loc>https://heihei.ru/Finland/destinations/</loc> <priority>0.6</priority> <changefreq>daily</changefreq> </url> </urlset>

There are just far more pages there.

To parse the sitemap we will use the xmlschema library, and to access the site, the requests library.

Let us prepare the test environment. We will create a virtual environment .

python -m venv venv

Let us activate it.

On Windows the command looks like this:

.\venv\Scripts\activate

On Linux it is a bit different:

source ./venv/bin/activate

python -m pip install --upgrade pip
python -m pip install pytest requests xmlschema

Alternatively, the dependencies can be installed from requirements.txt

pytest==8.3.5 requests==2.32.3 xmlschema==3.4.3

python -m pip install -r requirements.txt

# site_map.py import xml.etree.ElementTree as ET def get_urls() -> list: tree = ET.parse("sitemap.xml") root = tree.getroot() urls = [] for child in root: for gc in child: tag = gc.tag if "loc" in tag: urls.append(gc.text) return urls

The test will be very concise. That is what we love PyTest for

# test_by_site_map.py import pytest import requests from site_map import get_urls endpoints = get_urls() @pytest.mark.parametrize('endpoint', endpoints) def test_status(endpoint): r = requests.get(endpoint, allow_redirects=False) assert r.status_code == 200

Let us run the test with the following command

python -m pytest -v --no-header tests\pytest\test_by_site_map.py

tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Georgia/holidays/easter] PASSED [ 0%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/destinations/] PASSED [ 0%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Italy/travel/] PASSED [ 0%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Germany/cities/dusseldorf/ferris_wheel/] PASSED [ 1%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Norway/destinations/from_oslo_to_paris.php] PASSED [ 1%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/suomen/suomen_har/002_SM2_5-19.php] PASSED [ 1%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/streetart/malminkartano.php] PASSED [ 1%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/holidays/laskiainen.php] PASSED [ 1%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/travel/SPb-Lappeenranta.php] PASSED [ 2%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Norway/holidays/may1.php] PASSED [ 3%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/suomen/suomen_har/002_SM2_5-02.php] PASSED [ 3%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland//news/2021_06_12_green_mayor_candidate_corruption.php] PASSED [ 4%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/cities/lappeenranta/fort.php] PASSED [ 4%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/fests/porvoo_valofest.php] PASSED [ 4%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/news/2019_05_23_vantaa_bicycle.php] PASSED [ 4%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Sweden/holidays/st_valentine.php] PASSED [ 4%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Sweden/museums/skansen/] PASSED [ 5%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Spain/cities/malaga/] PASSED [ 5%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland/beauty/] PASSED [ 5%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Finland//shops/kierratyskeskus.php] PASSED … tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Armenia/holidays/easter.php] PASSED [ 99%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/Norway/destinations/from_oslo_to_rome.php] PASSED [ 99%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/en/Finland/travel/HEL_gosleep.php] PASSED [ 99%] tests/pytest/test_by_site_map.py::test_status[https://heihei.ru/en/Finland/travel/ecolines_kamppi.php] PASSED [100%] =================================== 553 passed in 121.11s (0:02:01) ====================================

If Python cannot find the module with the code, you can try going to the directory that is the parent of tests - in our last example that is from_site_map - and add it to the system path.

In PowerShell the command will look like this:

$Env:Path += ";$pwd"

In Bash it is a bit different.

export PATH=$PATH:$(pwd)

Additional materials

Without eval

An example of creating a test similar to the example from the introduction without using eval()

import operator import pytest OPS = { ">": operator.gt, "<": operator.lt, "==": operator.eq, "!=": operator.ne, "<=": operator.le, ">=": operator.ge, "+": operator.add, "-": operator.sub, "*": operator.mul, "/": operator.truediv, } def evaluate(expr: str): # Try operators in order of decreasing length so "==" is matched before "=" for op in sorted(OPS.keys(), key=len, reverse=True): if op in expr: left, right = expr.split(op) left = int(left.strip()) right = int(right.strip()) return OPS[op](left, right) raise ValueError(f"Unsupported expression: {expr}") @pytest.mark.parametrize( "test_input, expected_result", [ ("99 > 99", False), ("100 > 99", True), ("-2 < 1", True), ("-0 == 0", True), ("7 - 3", 4), ("1 + 2", 3), ("30 / 5", 6), ("2 * 2", 5), ], ) def test_compare(test_input, expected_result): assert evaluate(test_input) == expected_result

… collected 8 items noeval.py::test_compare[99 > 99-False] PASSED [ 12%] noeval.py::test_compare[100 > 99-True] PASSED [ 25%] noeval.py::test_compare[-2 < 1-True] PASSED [ 37%] noeval.py::test_compare[-0 == 0-True] PASSED [ 50%] noeval.py::test_compare[7 - 3-4] PASSED [ 62%] noeval.py::test_compare[1 + 2-3] PASSED [ 75%] noeval.py::test_compare[30 / 5-6] PASSED [ 87%] noeval.py::test_compare[2 * 2-5] FAILED [100%] …

Article author: Arthur Isaev