Namespace packages in Python
Introduction
Before studying this article, make sure you are familiar with the topic
sys.path in Python
PEP 420
Namespace packages are a mechanism by which a single Python package is split across several
directories on disk.
Because there are several directories in which a namespace package resides, there is no single
__init__.py file. Therefore the __init__.py file
may not exist at all.
When a namespace package is imported, no code is executed by default.
Finding namespace packages
- Scanning all directories in sys.path
- Importing standard packages if they are found
- Importing standard modules if they are found
- If nothing is found, all suitable directories are attached to the namespace package
Consider the structure of an ordinary package from the article Packages in Python
demo_reader ├── compressed │ ├── bzipped.py │ ├── gzipped.py │ └── __init__.py ├── __init__.py ├── multireader.py └── util ├── __init__.py └── writer.py
Let us spread this package over two directories
project ├── path1 │ └── demo_reader │ ├── multireader.py │ └── util │ ├── __init__.py │ └── writer.py └── path2 └── demo_reader └── compressed ├── bzipped.py ├── gzipped.py └── __init__.py
>>> import sys >>> sys.path.extend(['./path1', './path2']) >>> import demo_reader >>> demo_reader.__path__
_NamespacePath(['/home/andrei/python/./path1/demo_reader', '/home/andrei/python/./path2/demo_reader'])
>>> import demo_reader.util >>> import demo_reader.compressed >>> demo_reader.util.__path__
['/home/andrei/python/./path1/demo_reader/util']
>>> demo_reader.compressed.__path__
['/home/andrei/python/./path2/demo_reader/compressed']
You can check that everything works as before the split by running the same commands
>>> import sys >>> sys.path.extend(['./path1', './path2']) >>> from demo_reader.multireader import MultiReader >>> r = MultiReader('test.bz2') >>> r.read() 'data compressed with bz2'
Executable Directories
Consider the project from this lesson
python ├── multi-reader-program │ └── demo_reader │ ├── compressed │ │ ├── bzipped.py │ │ ├── gzipped.py │ │ └── __init__.py │ ├── __init__.py │ ├── multireader.py │ └── util │ ├── __init__.py │ └── writer.py ├── test.bz2 └── test.gz
Trying to run the directory will fail
python multi-reader-program
/home/andrei/.pyenv/versions/3.9.5/bin/python: can't find '__main__' module in '/home/andrei/python/multi-reader-program'
Let us add to the root of the directory multi-reader-program the file __main__.py
# multi-reader-program/__main__.py print('executing multi-reader-program/__main__.py')
python multi-reader-program
executing multi-reader-program/__main__.py
When Python runs the file __main__.py it adds the directory containing this file to sys.path then from __main__.py any script from this directory can be imported.
Let us change the code of __main__.py
# multi-reader-program/__main__.py import sys from demo_reader.multireader import MultiReader filename = sys.argv[1] r = MultiReader(filename) print(r.read()) r.close()
Now the directory can be run
python multi-reader-program ./test.bz2
demo reader is being imported data compressed with bz2
Executable Zip Files
Consider the code from the previous paragraph
python ├── multi-reader-program │ ├── demo_reader │ │ ├── compressed │ │ │ ├── bzipped.py │ │ │ ├── gzipped.py │ │ │ └── __init__.py │ │ ├── __init__.py │ │ ├── multireader.py │ │ └── util │ │ ├── __init__.py │ │ └── writer.py │ └── __main__.py ├── test.bz2 └── test.gz
cd multi-reader-program
python -m zipfile -c ../multi-reader-program.zip *
cd ..
ls
multi-reader-program multi-reader-program.zip test.bz2 test.gz
python multi-reader-program.zip test.gz
demo reader is being imported data compressed with gz by Andrei
Executable Package
Consider the code from the previous paragraph, but move the directory demo_reader
out of the directory multi-reader-program
Add to the directory demo_reader
the file __main__.py
ch4-executable-package ├── demo_reader │ ├── compressed │ │ ├── bzipped.py │ │ ├── gzipped.py │ │ └── __init__.py │ ├── __init__.py │ ├── __main__.py │ ├── multireader.py │ └── util │ ├── __init__.py │ └── writer.py ├── test.bz2 └── test.gz
python demo_reader test.gz
Traceback (most recent call last): File "/home/andrei/.pyenv/versions/3.9.5/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/andrei/.pyenv/versions/3.9.5/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/andrei/python/demo_reader/__main__.py", line 7, in <module> from demo_reader.multireader import MultiReader ModuleNotFoundError: No module named 'demo_reader'
Running directories and packages is different
| python directory | python -m directory |
|---|---|
| A directory is run |
A package is run
The option -m tells Python to treat directory as a module |
| directory is added to sys.path | directory is treated as a package |
| directory/__main__.py is not contained in the package | directory/__main__.py is a submodule of the package directory |
In the current example, demo_reader has to be treated as a module
python -m demo_reader test.gz
demo reader is being imported data compressed with gz
Article author: Andrei Olegovich