Working with a REST API in Python

Introduction

In this article you will learn how to write requests to a REST API in Python 3.

If your goal is creating your own REST API, go to the article «Flask»

Before installing anything, make sure you are familiar with working in a virtual environment Python.

You can read about this in the article «Virtual environments in Python»

Preparation

Activate your virtual environment and install requests with the command

python3 -m pip install requests

Examine the list of installed modules

python3 -m pip list

Package Version ---------- --------- certifi 2020.6.20 chardet 3.0.4 idna 2.10 pip 20.2.3 requests 2.24.0 setuptools 50.3.1 urllib3 1.25.10 wheel 0.35.1

requests pulls in certifi, chardet, idna and urllib3 along with it

You can check where requests was installed in this environment with the command

python3 -m pip show requests

Name: requests Version: 2.24.0 Summary: Python HTTP for Humans. Home-page: https://requests.readthedocs.io Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: /home/andrei/python/virtualenvs/answerit_env/lib/python3.8/site-packages Requires: certifi, chardet, urllib3, idna Required-by:

GET

To make a GET request it is enough to import requests and execute requests.get

Create the file rdemo.py with the following content:

import requests r = requests.get('https:/xkcd.com/353/') print(r)

Run the script with the command

python3 rdemo.py

<Response [200]>

If you got 200, all is well. Let's change our code to find out what actions we can perform on the <Response [200]> object

dir(r) will give a list of available attributes and methods

import requests r = requests.get('https:/topbicycle.ru/b/stels_pilot_950_md_26.php') print(dir(r))

['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_content', '_content_consumed', '_next', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'next', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']

More detailed information can be obtained by replacing dir(r) with help(r)

If you are interested, read the article requests help

From this document you can learn that the http status is contained in the attribute status_code

import requests r = requests.get('https:/topbicycle.ru/b/stels_pilot_950_md_26.php') print(r.status_code)

If everything went well, you will get

200

ok returns True if the response is not 4XX or 5XX

headers returns the response header

Also from this document you can learn that text returns the response body in unicode format, while content is the response body in bytes

Change the code to

import requests r = requests.get('https:/xkcd.com/353/') print("content:") print("-----") print(r.content) print("-----") print("text:") print("-----") print(r.text)

And study the difference

import requests r = requests.get(https://topbicycle.ru/b/img/stels_pilot_950_MD_26.jpg") with open("bike.jpg", "wb") as f: f.write(r.content)

If everything went well, the following photo will appear in your folder

GET with parameters

To test your REST API skills you can use the website httpbin.org

It will send your request back to you.

Let's change the code rdemo.py :

Parameters can be written right after the url following a question mark, but it is more reliable to put them in a separate dictionary.

import requests payload = {'page': 2, 'count': 25} r = requests.get(https://httpbin.org/get", params=payload) print(f"url: {r.url} \n\ntext: \n {r.text}")

python3 rdemo.py

url: https://httpbin.org/get?page=2&count=25 text: { "args": { "count": "25", "page": "2" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "httpbin.org", "User-Agent": "python-requests/2.24.0", "X-Amzn-Trace-Id": "Root=1-5f8c2d50-4f48f16c0991ad0e6e45676e" }, "origin": "87.92.8.47", "url": "https://httpbin.org/get?page=2&count=25" }

POST

To send and test a POST request, make a few small changes to the code.

Obviously, GET must be replaced with POST, and params must be replaced with data

import requests payload = {'website': 'heihei.ru', 'established': 2018} r = requests.post('https://httpbin.org/post', data=payload) print(f"url: {r.url} \n\ntext: \n {r.text}")

url: https://httpbin.org/post text: { "args": {}, "data": "", "files": {}, "form": { "established": "2018", "website": "heihei.ru" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "34", "Content-Type": "application/x-www-form-urlencoded", "Host": "httpbin.org", "User-Agent": "python-requests/2.24.0", "X-Amzn-Trace-Id": "Root=1-5f8c30ac-6b32899f073f9df4055c55c4" }, "json": null, "origin": "87.92.8.47", "url": "https://httpbin.org/post" }

The responses we received from httpbin came in json format. When working with json it is often convenient to use the built-in .json() method

In this case the data is written into a dictionary and is very easy to access.

Note the "form" data that was sent is returned in this field.

Let's change the code so that only this data is printed to the screen

import requests payload = {'website': 'heihei.ru', 'established': 2018} r = requests.post('https://httpbin.org/post', data=payload) r_dict = r.json() print(r_dict['form'])

python3 rdemo.py

{'established': '2018', 'website': 'heihei.ru'}

PUT

Everything is the same as with POST, just replace post with put

Let's look at an example where you need to send json in the request body and also use an existing token for authorization

import requests import json url = "http://eth1.ru" new_access_token = sdlfjsljglkjfd;lkgjdlkhjlkjgdlkhjlkdjglkj body = """[{"id":"1234abc","name":"andrei","website":"eth1.ru"}]""" payload = json.loads(body) json_headers = {"Content-type": "application/json", "Authorization": "Bearer %s" %new_access_token } r = requests.put(url, headers=json_headers, json=payload, verify=False)

Pay attention to the following points

• When sending JSON you must specify the header 'Content-type': 'application/json'
• The token is passed using Bearer
• You can easily convert data to JSON with json.loads()

Handling the response

Let's look at techniques that will come in handy when working with the received data

Suppose data has been received in the format json.

We need to extract from it the token, which is stored in access_token

To get it, use the json() method, which returns a dictionary (<class 'dict'>).

And from this dictionary get the token by the access_token key

r_dict = r.json() access_token = r_dict["access_token"]

json.dumps

If you want to examine the received json right away, you can use the method dumps()

import json print(json.dumps(response.json(), indent=4))

If you got a dict rather than json, json.dumps must be used like this:

import json print(json.dumps(resp.dict, indent=4, sort_keys=True))

sort_keys is optional. I am just showing it as an example, to demonstrate that the keys can be sorted.

Extracting part of the response

Often it is not the whole response that is of interest, but only a part. Read about how to extract it correctly from the response in the article

«How to access nested objects in json»

Authentication

Let's look at basic authentication on the httpbin website

Make up any username and a password for it.

I made up andrey with the password heihei

Go to httpbin.org . Make sure the address bar contains basic-auth/andrey/heihei or the login and password you made up.

Enter your login and password

Make sure the authentication succeeded

Now let's do the same authentication with Python

Create the file auth_demo.py with the following code

import requests r = requests.get('https://httpbin.org/basic-auth/andrey/heihei', auth=('andrey', 'heihei') ) print(r.text)

python3 auth_demo.py

{ "authenticated": true, "user": "andrey" }

The response matches what we already received in the browser

Make the same request, but with a wrong password. Make sure that text contains nothing. Replace print(r.text) with print(r) and make sure the object received is

<Response [401]>

Delay

It is often necessary to limit the response waiting time. This can be done with the timeout parameter

Go to httpbin.org the section - / #/ Dynamic_data / delete_delay__delay_ and study the documentation - if you make a request to this url you can set the time after which the response will be sent.

Create the file timeout_demo.py with the following content

import requests r = requests.get('https://httpbin.org/delay/1', timeout=3) print(r)

The delay is one second. And the response can be awaited for up to three seconds.

python3 timeout_demo.py

<Response [200]>

Change the code so that the response arrives definitely later than our three-second timeout.

import requests r = requests.get('https://httpbin.org/delay/7', timeout=3) print(r)

The delay is seven seconds. And the response can still be awaited for only up to three seconds.

python3 timeout_demo.py

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/usr/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.8/http/client.py", line 268, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.8/socket.py", line 669, in readinto return self._sock.recv_into(b) File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into raise timeout("The read operation timed out") socket.timeout: The read operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise raise value File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 423, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 330, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='httpbin.org', port=443): Read timed out. (read timeout=3) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "timeout_demo.py", line 4, in <module> r = requests.get('https://httpbin.org/delay/7', timeout=3) File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='httpbin.org', port=443): Read timed out. (read timeout=3)

If you are not thrilled by such exception handling, change the code using try except

import requests try: r = requests.get('https://httpbin.org/delay/7', timeout=3) except (requests.exceptions.Timeout, requests.exceptions.ConnectionError ) as err: print('Response is taking too long.') else: print(r)

python3 timeout_demo.py

Response is taking too long.

Article author: Andrei Olegovich