glob Python
Introduction
In this article you will learn how to work with several files at once with glob.
The glob module finds all path names matching the specified pattern, according to the rules
used by the
Unix
, although the results are returned in an arbitrary order.
Tilde expansion is not performed, but *, ? and character ranges expressed with [] are matched correctly.
This is done with the os.scandir() and fnmatch.fnmatch() functions together, not by actually calling a subshell.
Official documentation
Example
Suppose that we are in the directory glob_ex:
glob_ex/ └── sample_files ├── it.txt ├── testing.txt └── travel.txt
# glob_ex.py import glob files = glob.glob("./sample_files/*.txt") print(files)
python glob_ex.py
['./sample_files/it.txt', './sample_files/travel.txt']
Searching by pattern
Examples of other patterns for searching
… files = glob.glob("./sample_files/?.txt") print(files) files = glob.glob("./sample_files/??.txt") print(files) files = glob.glob("./sample_files/t*.txt") print(files)
['./sample_files/it.txt'] ['./sample_files/it.txt'] ['./sample_files/testing.txt', './sample_files/travel.txt']
… # starts with character from the range files = glob.glob("./sample_files/[it]*.txt") print(files) files = glob.glob("./sample_files/[ghi]*.txt") print(files) files = glob.glob("./sample_files/[rst]*.txt") print(files) files = glob.glob("./sample_files/[abc]*.txt") print(files)
['./sample_files/it.txt', './sample_files/testing.txt', './sample_files/travel.txt'] ['./sample_files/it.txt'] ['./sample_files/testing.txt', './sample_files/travel.txt'] []
… # starts with character not from the range files = glob.glob("./sample_files/[!i]*.txt") print(files) files = glob.glob("./sample_files/[!t]*.txt") print(files) files = glob.glob("./sample_files/[!abc]*.txt") print(files)
['./sample_files/testing.txt', './sample_files/travel.txt'] ['./sample_files/it.txt'] ['./sample_files/it.txt', './sample_files/testing.txt', './sample_files/travel.txt']
# glob_ex.py import glob files = glob.glob("./sample_files/*.txt") for file in files: print(file) with open(file, "r") as f: lines = f.readlines() for line in lines: print(line, end = "")
python glob_ex.py
./sample_files/it.txt beget.com kaspersky.com ./sample_files/testing.txt www.eth1.ru www.otus.ru ./sample_files/travel.txt aviasales.com booking.com
Searching in several directories
Let us update the project structure
glob_ex/ ├── glob_ex.py ├── heihei │ ├── Finland.html │ ├── Spain.html │ └── Sweden.html └── topbicycle ├── forward.html ├── stark.html └── stels.html
To search in several directories inside the root at once you need to use **.
If your Python version is 3.10 or higher I recommend specifying
the root directory with root_dir
import glob files = glob.glob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/" ) print(files)
['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']
root_dir must be specified according to the environment in which Python runs.
In this example Python runs in
WSL2
, so /mnt is placed before /c.
/mnt/c/Users/Andrei/glob_ex/
On Git Bash the path will start with / right away
/c/Users/Andrei/glob_ex/
In a full Linux most likely from /home
/home/andrei/glob_ex/
On Windows , for example in PowerShell the same path will look like
C:\Users\Andrei\glob_ex
Searching in nested directories
Let us complicate the project structure by adding nested directories
glob_ex/ ├── glob_ex.py ├── heihei │ ├── Armenia │ │ ├── .hidden.html │ │ ├── holidays │ │ │ ├── easter.html │ │ │ └── sargis.html │ │ └── index.html │ ├── Finland.html │ ├── Spain.html │ └── Sweden.html └── topbicycle ├── forward.html ├── stark.html └── stels.html
If you run the same script now it will not find files in the new Armenia subdirectory.
['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']
The parameter recursive controls the search in nested directories. If you set recursive=True the search will be carried out in all subdirectories.
import glob files = glob.glob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True ) print(files)
['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'heihei/Armenia/index.html', 'heihei/Armenia/holidays/easter.html', 'heihei/Armenia/holidays/sargis.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']
Now the script finds all files from the Armenia subdirectory matching the pattern.
Searching for hidden files
If you have Python 3.11 or higher, you can use the option include_hidden,
with which you can find the file in the Armenia directory
.hidden.html
I remind you that the new project structure can be studied
here
You can read about how to install a fresh Python version on
Linux
or
WSL
in the article
«Installing Python on Linux»
import glob files = glob.glob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(files)
['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'heihei/Armenia/.hidden.html', 'heihei/Armenia/index.html', 'heihei/Armenia/holidays/easter.html', 'heihei/Armenia/holidays/sargis.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']
With all these options the same patterns as before can be applied.
import glob files = glob.glob("**/[st]*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(files)
['heihei/Armenia/holidays/sargis.html', 'topbicycle/stark.html', 'topbicycle/stels.html']
import glob files = glob.glob("**/[St]*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(files)
['heihei/Spain.html', 'heihei/Sweden.html']
iglob
For working with large amounts of data the method may be useful iglob, which returns an iterator
import glob globs = glob.iglob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(globs) print("type(globs): ", type(globs)) print("globs.__next__: ", globs.__next__) print("type(globs.__next__): ", type(globs.__next__)) print(globs.__next__()) print(globs.__next__()) # another way to call next() print(next(globs)) print(next(globs))
<itertools.chain object at 0x7fce30e36ad0> type(globs): <class 'itertools.chain'> globs.__next__: <method-wrapper '__next__' of itertools.chain object at 0x7fce30e36ad0> type(globs.__next__): <class 'method-wrapper'> heihei/Finland.html heihei/Spain.html heihei/Sweden.html heihei/Armenia/.hidden.html
You can go through all files with
a for loop
Let us find all .py files in the directory python. For greater clarity of output we use the function
enumerate()
import glob globs = glob.iglob("**/*.py", root_dir="/mnt/c/Users/Andrei/python/", recursive=True, include_hidden=True ) for i, file in enumerate(globs, 2): print(i, file, sep=': ')
2: copy_ex.py 3: dict_ex.py 4: dir_ex.py 5: pass_arg.py 6: random_gen.py 7: random_gen2.py 8: tcompl_ex.py 9: glob_ex/glob_ex.py 10: glob_ex/glob_ex1.py 11: glob_ex/glob_ex3.py 12: glob_ex/iglob_ex.py 13: Python-3.11.3/python-config.py 14: Python-3.11.3/python.exe-gdb.py 15: Python-3.11.3/setup.py 16: Python-3.11.3/build/lib.linux-x86_64-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py 17: Python-3.11.3/Doc/conf.py … 2934: venv/Lib/site-packages/setuptools/_vendor/tomli/__init__.py 2935: venv/Lib/site-packages/_distutils_hack/override.py 2936: venv/Lib/site-packages/_distutils_hack/__init__.py
Article author: Andrei Olegovich