Namespace packages in Python

Introduction

Before studying this article, make sure you are familiar with the topic sys.path in Python

PEP 420

Namespace packages are a mechanism by which a single Python package is split across several directories on disk.

Because there are several directories in which a namespace package resides, there is no single __init__.py file. Therefore the __init__.py file may not exist at all.

When a namespace package is imported, no code is executed by default.

Finding namespace packages

Consider the structure of an ordinary package from the article Packages in Python

demo_reader ├── compressed │ ├── bzipped.py │ ├── gzipped.py │ └── __init__.py ├── __init__.py ├── multireader.py └── util ├── __init__.py └── writer.py

Let us spread this package over two directories

project ├── path1 │ └── demo_reader │ ├── multireader.py │ └── util │ ├── __init__.py │ └── writer.py └── path2 └── demo_reader └── compressed ├── bzipped.py ├── gzipped.py └── __init__.py

>>> import sys >>> sys.path.extend(['./path1', './path2']) >>> import demo_reader >>> demo_reader.__path__

_NamespacePath(['/home/andrei/python/./path1/demo_reader', '/home/andrei/python/./path2/demo_reader'])

>>> import demo_reader.util >>> import demo_reader.compressed >>> demo_reader.util.__path__

['/home/andrei/python/./path1/demo_reader/util']

>>> demo_reader.compressed.__path__

['/home/andrei/python/./path2/demo_reader/compressed']

You can check that everything works as before the split by running the same commands

>>> import sys >>> sys.path.extend(['./path1', './path2']) >>> from demo_reader.multireader import MultiReader >>> r = MultiReader('test.bz2') >>> r.read() 'data compressed with bz2'

Executable Directories

Consider the project from this lesson

python ├── multi-reader-program │ └── demo_reader │ ├── compressed │ │ ├── bzipped.py │ │ ├── gzipped.py │ │ └── __init__.py │ ├── __init__.py │ ├── multireader.py │ └── util │ ├── __init__.py │ └── writer.py ├── test.bz2 └── test.gz

Trying to run the directory will fail

python multi-reader-program

/home/andrei/.pyenv/versions/3.9.5/bin/python: can't find '__main__' module in '/home/andrei/python/multi-reader-program'

Let us add to the root of the directory multi-reader-program the file __main__.py

# multi-reader-program/__main__.py print('executing multi-reader-program/__main__.py')

python multi-reader-program

executing multi-reader-program/__main__.py

When Python runs the file __main__.py it adds the directory containing this file to sys.path then from __main__.py any script from this directory can be imported.

Let us change the code of __main__.py

# multi-reader-program/__main__.py import sys from demo_reader.multireader import MultiReader filename = sys.argv[1] r = MultiReader(filename) print(r.read()) r.close()

Now the directory can be run

python multi-reader-program ./test.bz2

demo reader is being imported data compressed with bz2

Executable Zip Files

Consider the code from the previous paragraph

python ├── multi-reader-program │ ├── demo_reader │ │ ├── compressed │ │ │ ├── bzipped.py │ │ │ ├── gzipped.py │ │ │ └── __init__.py │ │ ├── __init__.py │ │ ├── multireader.py │ │ └── util │ │ ├── __init__.py │ │ └── writer.py │ └── __main__.py ├── test.bz2 └── test.gz

cd multi-reader-program
python -m zipfile -c ../multi-reader-program.zip *
cd ..
ls

multi-reader-program multi-reader-program.zip test.bz2 test.gz

python multi-reader-program.zip test.gz

demo reader is being imported data compressed with gz by Andrei

Executable Package

Consider the code from the previous paragraph, but move the directory demo_reader out of the directory multi-reader-program

Add to the directory demo_reader the file __main__.py

ch4-executable-package ├── demo_reader │ ├── compressed │ │ ├── bzipped.py │ │ ├── gzipped.py │ │ └── __init__.py │ ├── __init__.py │ ├── __main__.py │ ├── multireader.py │ └── util │ ├── __init__.py │ └── writer.py ├── test.bz2 └── test.gz

python demo_reader test.gz

Traceback (most recent call last): File "/home/andrei/.pyenv/versions/3.9.5/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/andrei/.pyenv/versions/3.9.5/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/andrei/python/demo_reader/__main__.py", line 7, in <module> from demo_reader.multireader import MultiReader ModuleNotFoundError: No module named 'demo_reader'

Running directories and packages is different

python directory python -m directory
A directory is run A package is run

The option -m tells Python to treat directory as a module
directory is added to sys.path directory is treated as a package
directory/__main__.py is not contained in the package directory/__main__.py is a submodule of the package directory

In the current example, demo_reader has to be treated as a module

python -m demo_reader test.gz

demo reader is being imported data compressed with gz

Article author: Andrei Olegovich