Packages in Python
Introduction
Before studying this article, make sure you are familiar with the topic sys.path in Python
By default the main tool for organizing programs is the module.
Usually a module is a file with Python code and a .py extension
One module can be loaded into another module or into the REPL using import.
A module, like everything else, has a representation as an object.
A package is a type of module that can contain other modules as well as other packages.
Let's look at the example of urllib
>>> import urllib
>>> import urllib.request
>>> type(urllib)
<class 'module'>
>>> type(urllib.request)
<class 'module'>
Both urllib and urllib.request have the type module
With this kind of import you cannot call request without specifying the parent module.
Let's look at another way of importing
>>> from urllib import request
>>> request
<module 'urllib.request' from 'C:\\Users\\Andrei\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\urllib\\request.py'>
You can see that request is a child module of urllib
__path__
The difference between urllib and urllib.request can be seen from the fact that request has no __path__ attribute
>>> urllib.__path__
['C:\\Users\\Andrei\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\urllib']
Starting with Python 3.3+ __path__ is a list before that it was simply a string
>>> urllib.request.__path__
Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'urllib.request' has no attribute '__path__'
urllib is a package. Packages are usually directories, modules are usually plain files.
sys.path
Before creating your own packages, study the article
sys.path
From it you will learn how to make a module visible to Python
using
sys.path.append
or using
PYTHONPATH
Create a package
First of all you need to add the working directory to PYTHONPATH
export PYTHONPATH=/home/andrei/packages/
Note that it is the directory that contains the package that is added to PYTHONPATH, and not the package directory itself.
Inside this directory let's create the following project structure
packages └── pac ├── app │ └── double_sum.py └── lib └── regular_sum.py
The script reqular_sum.py adds two numbers
# regular_sum.py def my_sum(a: float, b: float) -> float: return a + b if __name__ == "__main__": my_sum(4, 7)
The script double_sum.py adds two numbers using the function imported from regular_sum.py - the function my_sum() - and doubles this result
# double_sum from regular_sum import my_sum def double_sum(a: float, b: float) -> float: s = my_sum(a, b) return s * 2 if __name__ == "__main__": print(double_sum(7, 11)) # expected result: 36
If you run the script double_sum.py right away, you will get an error message
python double_sum.py
Traceback (most recent call last): File "/home/andrei/packages/pac/app/double_sum.py", line 1, in <module> from regular_sum import my_sum ModuleNotFoundError: No module named 'regular_sum'
You need to change the import by specifying from which directory to take the function. Nested directories are specified with a dot.
from pac.lib.regular_sum import my_sum
Full code:
# double_sum from pac.lib.regular_sum import my_sum def double_sum(a: float, b: float) -> float: s = my_sum(a, b) return s * 2 if __name__ == "__main__": print(double_sum(7, 11)) # expected result: 36
python double_sum.py
36
Example with os.path.splitext
You can read more about the method os.path in the article «The os module in Python»
import os my_file = "demo.custom" filename = os.path.splitext(my_file)[0] extension = os.path.splitext(my_file)[1] print(filename) print(extension)
demo
.custom
__init__.py
In
modern
Python
the actions described above are enough to create a package.
Nevertheless it is recommended to add files to the directories that are part of the package
__init__.py
. They can be either empty or contain some settings.
Why this is useful:
- Compatibility with Python below 3.3 is preserved
- It will be clear that a package is meant
- In these files you can set settings, for example for __all__
With __init__.py files the structure of the previous project will look like this
pac ├── app │ ├── double_sum.py │ └── __init__.py ├── __init__.py └── lib ├── __init__.py └── regular_sum.py
Example: demoreader
Let's look at an example of using __init__.py
mkdir demo_reader touch demo_reader/__init__.py python >>> import demo_reader >>> type(demo_reader)
<class 'module'>
>>> demo_reader.__file__
'/home/andrei/demo_reader/__init__.py'
Let's make it so that __init__.py informs us every time that the package has been imported
echo 'print("demo reader is being imported")' >> demo_reader/__init__.py
python >>> import demo_reader
demo reader is being imported
Let's add to the package demo_reader the file multireader.py
demo_reader ├── __init__.py └── multireader.py
# demo_reader/__init__.py print("demo reader is being imported")
# demo_reader/multireader.py class MultiReader: def __init__(self, filename): self.filename = filename self.f = open(filename, 'rt') def close(self): self.f.close() def read(self): return self.f.read()
python >>> import demo_reader.multireader
demo reader is being imported
>>> r = demo_reader.multireader.MultiReader('demo_reader/__init__.py') >>> r.read()
'# demo_reader/__init__.py\n\nprint("demo reader is being imported")\n'
>>> r.close()
Let's add the ability to read files compressed with bz2 and gzip
demo_reader ├── compressed │ ├── bzipped.py │ ├── gzipped.py │ └── __init__.py ├── __init__.py └── multireader.py
# demo_reader/compressed/gzipped.py import gzip import sys opener = gzip.open # Alias for gzip.open # Decompresses during read if __name__ == '__main__': # Use gzip to create compressed file f = gzip.open(sys.argv[1], mode='wt') # Join to space-separaded string f.write(' '.join( sys.argv[2:]) # The data to compress ) f.close()
python -m demo_reader.compressed.gzipped test.gz data compressed with gz
The option -m means that a module needs to be run
demo_reader.compressed.gzipped is the module name. It must be in the format
FQMN - Fully-qualified module name, that is, it must contain all the required directories separated by dots.
test.gz is argv[1], that is, the name of the file being written to
Everything that follows is argv[2:], in this case argv[2], argv[3], argv[4] and argv[5]
- that is, the data that will be written to the file
A similar script for bz2
# demo_reader/compressed/bzipped.py import bz2 import sys opener = bz2.open if __name__ == '__main__': f = bz2.open(sys.argv[1], mode='wt') f.write(' '.join(sys.argv[2:])) f.close()
python -m demo_reader.compressed.bzipped test.bz2 data compressed with bz2
As a result we now have two files test.gz and test.bz2
All modules and packages can be imported
python >>> import demo_reader demo reader is being imported >>> import demo_reader.multireader >>> import demo_reader.compressed >>> import demo_reader.compressed.gzipped >>> import demo_reader.compressed.bzipped
Let's change the script multireader.py
to use the new modules in it.
About the
get()
method, read in the article
dictionaries in Python
# demo_reader/multireader.py import os from demo_reader.compressed import bzipped, gzipped extension_map = { '.bz2': bzipped.opener, '.gz': gzipped.opener } class MultiReader: def __init__(self, filename): extension = os.path.splitext(filename)[1] opener = extension_map.get(extension, open) self.f = opener(filename, 'rt') def close(self): self.f.close() def read(self): return self.f.read()
python >>> from demo_reader.multireader import MultiReader
demo reader is being imported
>>> r = MultiReader('test.bz2') >>> r.read()
'data compressed with bz2'
>>> r.close() >>> r = MultiReader('test.gz') >>> r.read()
'data compressed with gzip'
>>> r.close() >>> r = MultiReader('demo_reader/__init__.py')
>>> r.read()
'# demo_reader/__init__.py\n\nprint("demo reader is being imported")\n'
>>> r.close()
Absolute import
Both examples below use an absolute import. The full name of the package and the name of the module are specified.
import demo_reader.compressed.bzipped from demo_reader.compressed import bzipped
Relative import
Both examples below use a relative import. It can only be used inside a package.
You can specify the name of the package (subpackage) or simply specify the path.
from ..module_name import name from ..import name
The advantages of a relative import are less code and the ability to change the package name without changing
the code of the modules.
Nevertheless standard practice is to avoid relative imports everywhere where this is
possible. With the help of modern
IDE
, such as
PyCharm
you can easily refactor code in a semi-automatic mode. This makes the advantages of a relative
import insignificant.
Example with a relative import
Let's change our code to reduce duplication and at the same time show both types of import.
Let's add the directory
util
and in it the files
__init__.py
and
writer.py
demo_reader ├── compressed │ ├── bzipped.py │ ├── gzipped.py │ └── __init__.py ├── __init__.py ├── multireader.py └── util ├── __init__.py └── writer.py
# demo_reader/util/writer.py import sys def main(opener): f = opener(sys.argv[1], mode='wt') f.write(' '.join(sys.argv[2:])) f.close()
Now
bzipped.py
and
gzipped.py
can use
writer.py
For this
bzipped.py
imports
writer.py
through an absolute path, while
gzipped.py
through a relative one
# demo_reader/compressed/bzipped.py import bz2 from demo_reader.util import writer opener = bz2.open if __name__ == '__main__': writer.main(opener)
Note how the function is called main - with a dot after the module name
# demo_reader/compressed/gzipped.py import gzip from ..util import writer opener = gzip.open if __name__ == '__main__': writer.main(opener)
Using the example of
demo_reader/compressed/bzipped.py
Let's look at what absolute and relative import paths look like
| Relative | Absolute |
|---|---|
| from . import name | from demo_reader.compressed import name |
| from .. import name | from demo_reader import name |
| from ..util import name | from demo_reader.util import name |
| from .. import util | from demo_reader import util |
__all__
__all__ is a module attribute. It is responsible for what will be imported if someone wants to do
from module import *
If __all__ is not set, then after import * all public names will be imported
__all__ must be a list of strings each element of which is a name to import
Let's edit the file __init__.py in the directory compressed
from demo_reader.compressed.bzipped import opener as bz2_opener from demo_reader.compressed.gzipped import opener as gzip_opener
Without __all__
python >>> from pprint import pprint >>> pprint(locals())
{'__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__doc__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__name__': '__main__', '__package__': None, '__spec__': None, 'pprint': <function pprint at 0x7fe88894f3a0>}
>>> from demo_reader.compressed import *
demo reader is being imported
>>> pprint(locals())
{'__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__doc__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__name__': '__main__', '__package__': None, '__spec__': None, 'bz2_opener': <function open at 0x7fe888921ee0>, 'bzipped': <module 'demo_reader.compressed.bzipped' from '/home/avorotyn/python/lessons/pluralsight/organizing_larger_programs/chapter3/demo_reader/compressed/bzipped.py'>, 'gzip_opener': <function open at 0x7fe8888d34c0>, 'gzipped': <module 'demo_reader.compressed.gzipped' from '/home/avorotyn/python/lessons/pluralsight/organizing_larger_programs/chapter3/demo_reader/compressed/gzipped.py'>, 'pprint': <function pprint at 0x7fe88894f3a0>}
With the help of __all__ you can import only bz2_opener and gzip_opener
# demo_reader/compressed/__init__.py from demo_reader.compressed.bzipped import opener as bz2_opener from demo_reader.compressed.gzipped import opener as gzip_opener __all__ = ['bz2_opener', 'gzip_opener']
python >>> from pprint import pprint >>> pprint(locals())
{'__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__doc__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__name__': '__main__', '__package__': None, '__spec__': None, 'pprint': <function pprint at 0x7fe88894f3a0>}
>>> from demo_reader.compressed import *
demo reader is being imported
>>> pprint(locals())
{'__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__doc__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__name__': '__main__', '__package__': None, '__spec__': None, 'bz2_opener': <function open at 0x7fe888921ee0>, 'gzip_opener': <function open at 0x7fe8888d34c0>, 'pprint': <function pprint at 0x7fe88894f3a0>}
>>> bz2_opener
<function open at 0x7f3db4c0fee0>
>>> gzip_opener
<function open at 0x7f3db4bc34c0>
Full code
Creating archives with
python -m demo_reader.compressed.bzipped test.bz2 data compressed with bz2
python -m demo_reader.compressed.gzipped test.gz data compressed with gz
In this variant it will give a warning
/home/andrei/.pyenv/versions/3.9.5/lib/python3.9/runpy.py:127: RuntimeWarning: 'demo_reader.compressed.bzipped' found in sys.modules after import of package 'demo_reader.compressed', but prior to execution of 'demo_reader.compressed.bzipped'; this may result in unpredictable behaviour warn(RuntimeWarning(msg))
The archive will nevertheless be created, I am still thinking about the solution.
Structure
demo_reader ├── compressed │ ├── bzipped.py │ ├── gzipped.py │ └── __init__.py ├── __init__.py ├── multireader.py └── util ├── __init__.py └── writer.py
# demo_reader/compressed/bzipped.py import bz2 from demo_reader.util import writer opener = bz2.open if __name__ == '__main__': writer.main(opener)
# demo_reader/compressed/gzipped.py import gzip from ..util import writer opener = gzip.open if __name__ == '__main__': writer.main(opener)
# demo_reader/compressed/__init__.py from demo_reader.compressed.bzipped import opener as bz2_opener from demo_reader.compressed.gzipped import opener as gzip_opener __all__ = ['bz2_opener', 'gzip_opener']
# demo_reader/__init__.py print("demo reader is being imported")
# demo_reader/multireader.py import os from demo_reader.compressed import bzipped, gzipped extension_map = { '.bz2': bzipped.opener, '.gz': gzipped.opener } class MultiReader: def __init__(self, filename): extension = os.path.splitext(filename)[1] opener = extension_map.get(extension, open) self.f = opener(filename, 'rt') def close(self): self.f.close() def read(self): return self.f.read()
# demo_reader/util/__init__.py
# demo_reader/util/writer.py import sys def main(opener): f = opener(sys.argv[1], mode='wt') f.write(' '.join(sys.argv[2:])) f.close()
python -m demo_reader.compressed.bzipped test.bz2 data compressed with bz2 by Andrei
python >>> from demo_reader.multireader import MultiReader
demo reader is being imported
>>> r = MultiReader('test.bz2') >>> r.read()
'data compressed with bz2 by Andrei'
Article author: Andrei Olegovich