Pydantic

Introduction

Pydantic is a library with which you can parse data and perform validation.

A free translation of what they write about themselves + comments:

Data validation and settings management using python type annotations.

pydantic applies type annotations (type hints - see PEP 484 ) at runtime and provides user-friendly error messages when the data is invalid.

You can define what the data should be using pure, canonical python. Then you can perform validation using pydantic.

Pydantic uses a capability of modern Python. Make sure your version is at least 3.7

It is advisable to install the latest stable version of Python. If needed - read «Guide to installing Python on Linux»

Nevertheless, if you plan to exchange data in JSON format with Swagger or Open API, check the current library compatibility.

Usage example

Let's look at the script PydanticDemo.py

from dataclasses import dataclass from typing import Tuple from enum import Enum @dataclass class IceCreamMix: name: str flavor: str toppings: Tuple[str, ...] scoops: int def main(): ice_cream_mix = IceCreamMix( "PB&J", "peanut butter", ("strawberries", "sprinkles"), 2 ) print(ice_cream_mix) if __name__ == '__main__': main()

python PydanticDemo.py

IceCreamMix(name='PB&J', flavor='peanut butter', toppings=('strawberries', 'sprinkles'), scoops=2)

This script successfully demonstrates the ice cream type

Let's add a bit more OOP

from dataclasses import dataclass from typing import Tuple from enum import Enum class Flavor(str, Enum): chocolate = 'chocolate' vanilla = 'vanilla' strawberry = 'strawberry' mint = 'mint' coffeee = 'coffee' peanut_butter = 'peanut butter' class Topping(str, Enum): sprinkles = 'sprinkles' hot_fudge = 'hot fudge' cookies = 'cookies' brownie = 'brownie' whipped_cream = 'whipped cream' strawberries = 'strawberries' @dataclass class IceCreamMix: name: str flavor: Flavor toppings: Tuple[Topping, ...] scoops: int def main(): ice_cream_mix = IceCreamMix( "PB&J", Flavor.peanut_butter, (Topping.strawberries, Topping.sprinkles), 2 ) print(ice_cream_mix) if __name__ == '__main__': main()

$ python PydanticDemo.py

IceCreamMix(name='PB&J', flavor=<Flavor.peanut_butter: 'peanut butter'>, toppings=(<Topping.strawberries: 'strawberries'>, <Topping.sprinkles: 'sprinkles'>), scoops=2)

The script still works

What if we mistakenly choose a non-existent flavor or topping

def main(): ice_cream_mix = IceCreamMix( "PB&J", "smells like shit", (Topping.strawberries, 111), 2 )

python PydanticDemo.py

IceCreamMix(name='PB&J', flavor='smells like shit', toppings=(<Topping.strawberries: 'strawberries'>, 111), scoops=2)

The script does not notice the catch.

To validate the data automatically, install pydantic and make just one change to the first line

from pydantic.dataclasses import dataclass

python PydanticDemo.py

Traceback (most recent call last): File "PydanticDemo.py", line 41, in <module> main() File "PydanticDemo.py", line 31, in main ice_cream_mix = IceCreamMix( File "<string>", line 7, in __init__ File "C:\Users\Andrei\python\pydantic\venv\lib\site-packages\pydantic\dataclasses.py", line 99, in _pydantic_post_init raise validation_error pydantic.error_wrappers.ValidationError: 2 validation errors for IceCreamMix flavor value is not a valid enumeration member; permitted: 'chocolate', 'vanilla', 'strawberry', 'mint', 'coffee', 'peanut butter' (type=type_error.enum; enum_values=[<Flavor.chocolate: 'chocolate'>, <Flavor.vanilla: 'vanilla'>, <Flavor.strawberry: 'strawberry'>, <Flavor.mint: 'mint'>, <Flavor.coffeee: 'coffee'>, <Flavor.peanut_butter: 'peanut butter'>]) toppings -> 1 value is not a valid enumeration member; permitted: 'sprinkles', 'hot fudge', 'cookies', 'brownie', 'whipped cream', 'strawberries' (type=type_error.enum; enum_values=[<Topping.sprinkles: 'sprinkles'>, <Topping.hot_fudge: 'hot fudge'>, <Topping.cookies: 'cookies'>, <Topping.brownie: 'brownie'>, <Topping.whipped_cream: 'whipped cream'>, <Topping.strawberries: 'strawberries'>])

pydantic did not let our code through. Let's examine the output in more detail

pydantic.error_wrappers.ValidationError: 2 validation errors for IceCreamMix

The number of errors and the class are indicated. This would help with debugging if we did not know in advance where the errors are and how many there are

flavor value is not a valid enumeration member; permitted: 'chocolate', 'vanilla', 'strawberry', 'mint', 'coffee', 'peanut butter' (type=type_error.enum; enum_values=[<Flavor.chocolate: 'chocolate'>, <Flavor.vanilla: 'vanilla'>, <Flavor.strawberry: 'strawberry'>, <Flavor.mint: 'mint'>, <Flavor.coffeee: 'coffee'>, <Flavor.peanut_butter: 'peanut butter'>])

Pydantic suggests the allowed values.

The same goes for the toppings, where 111 stands instead of an allowed value

toppings -> 1 value is not a valid enumeration member; permitted: 'sprinkles', 'hot fudge', 'cookies', 'brownie', 'whipped cream', 'strawberries' (type=type_error.enum; enum_values=[<Topping.sprinkles: 'sprinkles'>, <Topping.hot_fudge: 'hot fudge'>, <Topping.cookies: 'cookies'>, <Topping.brownie: 'brownie'>, <Topping.whipped_cream: 'whipped cream'>, <Topping.strawberries: 'strawberries'>])

Return the correct values for Flavor and Topping but replace scoops from 2 with '2'

def main(): ice_cream_mix = IceCreamMix( "PB&J", Flavor.peanut_butter, (Topping.strawberries, Topping.sprinkles), '2' )

python PydanticDemo.py

IceCreamMix(name='PB&J', flavor=<Flavor.peanut_butter: 'peanut butter'>, toppings=(<Topping.strawberries: 'strawberries'>, <Topping.sprinkles: 'sprinkles'>), scoops=2)

scoops is still equal to two

Pydantic supports type coercion

BaseModel

To access additional capabilities such as serialization (Serialization) and JSON support, let's use the BaseModel class

Just to remind you, at first we had

from dataclasses import dataclass

Then

from pydantic.dataclasses import dataclass

And now we need to make

from pydantic import BaseModel

And remove the @dataclass decorator before class IceCreamMix:

class IceCreamMix: it needs to be replaced with class IceCreamMix(BaseModel):

and also add attribute names to the code that creates the IceCreamMix class object that is name = "PB&J" flavor = Flavor.peanut_butter and so on

strawberries = 'strawberries' class IceCreamMix(BaseModel): name: str flavor: Flavor toppings: Tuple[Topping, ...] scoops: int def main(): ice_cream_mix = IceCreamMix( name = "PB&J", flavor = Flavor.peanut_butter, toppings = (Topping.strawberries, Topping.sprinkles), scoops = 2 )

python PydanticDemo.py

IceCreamMix(name='PB&J', flavor=<Flavor.peanut_butter: 'peanut butter'>, toppings=(<Topping.strawberries: 'strawberries'>, <Topping.sprinkles: 'sprinkles'>), scoops=2)

Everything works the same as before the changes.

Now you can output the result as JSON

print(ice_cream_mix.json())

python PydanticDemo.py

{"name": "PB&J", "flavor": "peanut butter", "toppings": ["strawberries", "brownie"], "scoops": 2}

Pay attention to the JSON you received above.

You can copy it, then if needed change it and create another object directly from JSON using the method parse_raw()

For example:

another_mix = IceCreamMix.parse_raw('{"name": "New mix", "flavor": "mint", "toppings": ["cookies", "hot fudge"], "scoops": 2}') print(another_mix.json())

{"name": "New mix", "flavor": "mint", "toppings": ["cookies", "hot fudge"], "scoops": 2}

If you accidentally make a mistake with an attribute value - pydantic will not let it pass

another_mix = IceCreamMix.parse_raw('{"name": "New mix", "flavor": "novichoke", "toppings": ["cookies", "hot fudge"], "scoops": 2}') print(another_mix.json())

python PydanticDemo.py

Traceback (most recent call last): File "/home/avorotyn/python/pydantic/PydanticDemo.py", line 45, in <module> main() File "/home/avorotyn/python/pydantic/PydanticDemo.py", line 40, in main another_mix = IceCreamMix.parse_raw('{"name": "New mix", "flavor": "novichoke", "toppings": ["cookies", "hot fudge"], "scoops": 2}') File "pydantic/main.py", line 543, in pydantic.main.BaseModel.parse_raw File "pydantic/main.py", line 520, in pydantic.main.BaseModel.parse_obj File "pydantic/main.py", line 362, in pydantic.main.BaseModel.__init__ pydantic.error_wrappers.ValidationError: 1 validation error for IceCreamMix flavor value is not a valid enumeration member; permitted: 'chocolate', 'vanilla', 'strawberry', 'mint', 'coffee', 'peanut butter' (type=type_error.enum; enum_values=[<Flavor.chocolate: 'chocolate'>, <Flavor.vanilla: 'vanilla'>, <Flavor.strawberry: 'strawberry'>, <Flavor.mint: 'mint'>, <Flavor.coffeee: 'coffee'>, <Flavor.peanut_butter: 'peanut butter'>])

ValidationError as JSON

Error messages can also be formatted as JSON. You need to import ValidationError from pydantic and use try: except

from pydantic import BaseModel, ValidationError …

def main(): try: ice_cream_mix = IceCreamMix( name = "PB&J", flavor = "spring", toppings = (Topping.strawberries, Topping.sprinkles), scoops = 2 ) except ValidationError as e: print(e.json())

python PydanticDemo.py

[ { "loc": [ "flavor" ], "msg": "value is not a valid enumeration member; permitted: 'chocolate', 'vanilla', 'strawberry', 'mint', 'coffee', 'peanut butter'", "type": "type_error.enum", "ctx": { "enum_values": [ "chocolate", "vanilla", "strawberry", "mint", "coffee", "peanut butter" ] } } ]

Bounds of allowed values

Suppose you want the number of scoops to be a required attribute with values from 0 to 5 excluding the bounds

from pydantic import BaseModel, ValidationError, Field …

strawberries = 'strawberries' class IceCreamMix(BaseModel): name: str flavor: Flavor toppings: Tuple[Topping, ...] scoops: int = Field(..., gt=0, lt=5)

python PydanticDemo.py

{"name": "PB&J", "flavor": "peanut butter", "toppings": ["strawberries", "brownie"], "scoops": 2}

2 scoops are set, so there are no errors.

Let's try 5 scoops

def main(): try: ice_cream_mix = IceCreamMix( name = "PB&J", flavor = "spring", toppings = (Topping.strawberries, Topping.sprinkles), scoops = 5 ) except ValidationError as e: print(e.json())

python PydanticDemo.py

[ { "loc": [ "scoops" ], "msg": "ensure this value is less than 5", "type": "value_error.number.not_lt", "ctx": { "limit_value": 5 } } ] Traceback (most recent call last): File "/home/avorotyn/python/pydantic/PydanticDemo.py", line 50, in <module> main() File "/home/avorotyn/python/pydantic/PydanticDemo.py", line 41, in main print(ice_cream_mix.json()) UnboundLocalError: local variable 'ice_cream_mix' referenced before assignment

Do not pay attention to the Traceback - the print could have been put inside try, but if everything is fine, the object is created and this error does not occur, and if it is not fine, pydantic catches the mismatch and outputs value_error

Validation using the decorator validator

Another useful way to set constraints - using @validator

It is used when you need to watch a single attribute

from pydantic import BaseModel, ValidationError, Field, validator …

class IceCreamMix(BaseModel): name: str flavor: Flavor toppings: Tuple[Topping, ...] scoops: int = Field(..., gt=0, lt=5) @validator('toppings') def check_toppings(cls, toppings): if len(toppings) > 4: raise ValueError('Too many toppings') return toppings

If you run this code with two toppings there will be no errors, so let's immediately add three more so that there are five in total.

def main(): try: ice_cream_mix = IceCreamMix( name = "PB&J", flavor = "spring", toppings = (Topping.strawberries, Topping.brownie,Topping.sprinkles,Topping.hot_fudge,Topping.whipped_cream),

python PydanticDemo.py

[ { "loc": [ "toppings" ], "msg": "Too many toppings", "type": "value_error" } ]

Now you can reduce the number of toppings to four and make sure there is no error.

@root_validator

It is used when you need to watch the whole model. For example, combinations of different attributes.

from pydantic import BaseModel, ValidationError, Field, validator, root_validator …

Create another class - Container

strawberries = 'strawberries' class Container(str, Enum): cup = 'cup' cone = 'cone' waffle_cone = 'waffle cone' class IceCreamMix(BaseModel): name: str flavor: Flavor

Let's set a condition: if the topping is hot_fudge then no cone can be given - only a cup

We will do the validation via @root_validator

@validator('toppings') def check_toppings(cls, toppings): if len(toppings) > 4: raise ValueError('Too many toppings') return toppings @root_validator def check_cone_toppings(cls, toppings): container = values.get('container') toppings = values.get('toppings') if container == Container.cone or container == Container.waffle_cone: if Topping.hot_fudge in toppings: raise ValueError('Cones cannot have hot fudge') return values def main(): try: ice_cream_mix = IceCreamMix( name = "PB&J", container = Container.waffle_cone, flavor = "spring",

You should still have the hot fudge topping left from the previous example, if not - add it and run

python PydanticDemo.py

[ { "loc": [ "__root__" ], "msg": "Cones cannot have hot fudge", "type": "value_error" } ]

Final code of the example

The brief overview of pydantic's capabilities has come to an end.

Thank you for your attention, below is the full code for this article.

from pydantic import BaseModel, \ ValidationError, Field, validator, root_validator from typing import Tuple from enum import Enum class Flavor(str, Enum): chocolate = 'chocolate' vanilla = 'vanilla' strawberry = 'strawberry' mint = 'mint' coffeee = 'coffee' peanut_butter = 'peanut butter' class Topping(str, Enum): sprinkles = 'sprinkles' hot_fudge = 'hot fudge' cookies = 'cookies' brownie = 'brownie' whipped_cream = 'whipped cream' strawberries = 'strawberries' class Container(str, Enum): cup = 'cup' cone = 'cone' waffle_cone = 'waffle cone' class IceCreamMix(BaseModel): name: str container: Container flavor: Flavor toppings: Tuple[Topping, ...] scoops: int = Field(..., gt=0, lt=5) @validator('toppings') def check_toppings(cls, toppings): if len(toppings) > 4: raise ValueError('Too many toppings') return toppings @root_validator def check_cone_toppings(cls, values): container = values.get('container') toppings = values.get('toppings') if container == Container.cone or container == Container.waffle_cone: if Topping.hot_fudge in toppings: raise ValueError('Cones cannot have hot fudge') return values def main(): try: ice_cream_mix = IceCreamMix( name="PB&J", container=Container.waffle_cone, flavor=Flavor.peanut_butter, # flavor='unknown flavour' toppings=(Topping.strawberries, Topping.brownie, Topping.sprinkles), # on validator # toppings=(Topping.strawberries, Topping.brownie, # Topping.sprinkles,Topping.cookies, Topping.sprinkles), # on root_validator # toppings=(Topping.strawberries, Topping.brownie, # Topping.sprinkles,Topping.hot_fudge), scoops=2 # scoops=5 ) print(ice_cream_mix.json()) except ValidationError as e: print(e.json()) if __name__ == '__main__': main()

The second part of the article

Article author: Andrei Olegovich

Banner image