glob Python

Introduction

In this article you will learn how to work with several files at once with glob.

The glob module finds all path names matching the specified pattern, according to the rules used by the Unix , although the results are returned in an arbitrary order.

Tilde expansion is not performed, but *, ? and character ranges expressed with [] are matched correctly. This is done with the os.scandir() and fnmatch.fnmatch() functions together, not by actually calling a subshell.

Official documentation

Example

Suppose that we are in the directory glob_ex:

glob_ex/ └── sample_files ├── it.txt ├── testing.txt └── travel.txt

# glob_ex.py import glob files = glob.glob("./sample_files/*.txt") print(files)

python glob_ex.py

['./sample_files/it.txt', './sample_files/travel.txt']

Searching by pattern

Examples of other patterns for searching

… files = glob.glob("./sample_files/?.txt") print(files) files = glob.glob("./sample_files/??.txt") print(files) files = glob.glob("./sample_files/t*.txt") print(files)

['./sample_files/it.txt'] ['./sample_files/it.txt'] ['./sample_files/testing.txt', './sample_files/travel.txt']

… # starts with character from the range files = glob.glob("./sample_files/[it]*.txt") print(files) files = glob.glob("./sample_files/[ghi]*.txt") print(files) files = glob.glob("./sample_files/[rst]*.txt") print(files) files = glob.glob("./sample_files/[abc]*.txt") print(files)

['./sample_files/it.txt', './sample_files/testing.txt', './sample_files/travel.txt'] ['./sample_files/it.txt'] ['./sample_files/testing.txt', './sample_files/travel.txt'] []

… # starts with character not from the range files = glob.glob("./sample_files/[!i]*.txt") print(files) files = glob.glob("./sample_files/[!t]*.txt") print(files) files = glob.glob("./sample_files/[!abc]*.txt") print(files)

['./sample_files/testing.txt', './sample_files/travel.txt'] ['./sample_files/it.txt'] ['./sample_files/it.txt', './sample_files/testing.txt', './sample_files/travel.txt']

# glob_ex.py import glob files = glob.glob("./sample_files/*.txt") for file in files: print(file) with open(file, "r") as f: lines = f.readlines() for line in lines: print(line, end = "")

python glob_ex.py

./sample_files/it.txt beget.com kaspersky.com ./sample_files/testing.txt www.eth1.ru www.otus.ru ./sample_files/travel.txt aviasales.com booking.com

Searching in several directories

Let us update the project structure

glob_ex/ ├── glob_ex.py ├── heihei │ ├── Finland.html │ ├── Spain.html │ └── Sweden.html └── topbicycle ├── forward.html ├── stark.html └── stels.html

To search in several directories inside the root at once you need to use **.

If your Python version is 3.10 or higher I recommend specifying the root directory with root_dir

import glob files = glob.glob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/" ) print(files)

['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']

root_dir must be specified according to the environment in which Python runs.

In this example Python runs in WSL2 , so /mnt is placed before /c.

/mnt/c/Users/Andrei/glob_ex/

On Git Bash the path will start with / right away

/c/Users/Andrei/glob_ex/

In a full Linux most likely from /home

/home/andrei/glob_ex/

On Windows , for example in PowerShell the same path will look like

C:\Users\Andrei\glob_ex

Searching in nested directories

Let us complicate the project structure by adding nested directories

glob_ex/ ├── glob_ex.py ├── heihei │ ├── Armenia │ │ ├── .hidden.html │ │ ├── holidays │ │ │ ├── easter.html │ │ │ └── sargis.html │ │ └── index.html │ ├── Finland.html │ ├── Spain.html │ └── Sweden.html └── topbicycle ├── forward.html ├── stark.html └── stels.html

If you run the same script now it will not find files in the new Armenia subdirectory.

['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']

The parameter recursive controls the search in nested directories. If you set recursive=True the search will be carried out in all subdirectories.

import glob files = glob.glob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True ) print(files)

['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'heihei/Armenia/index.html', 'heihei/Armenia/holidays/easter.html', 'heihei/Armenia/holidays/sargis.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']

Now the script finds all files from the Armenia subdirectory matching the pattern.

Searching for hidden files

If you have Python 3.11 or higher, you can use the option include_hidden, with which you can find the file in the Armenia directory .hidden.html

I remind you that the new project structure can be studied here

You can read about how to install a fresh Python version on Linux or WSL in the article «Installing Python on Linux»

import glob files = glob.glob("**/*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(files)

['heihei/Finland.html', 'heihei/Spain.html', 'heihei/Sweden.html', 'heihei/Armenia/.hidden.html', 'heihei/Armenia/index.html', 'heihei/Armenia/holidays/easter.html', 'heihei/Armenia/holidays/sargis.html', 'topbicycle/forward.html', 'topbicycle/stark.html', 'topbicycle/stels.html']

With all these options the same patterns as before can be applied.

import glob files = glob.glob("**/[st]*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(files)

['heihei/Armenia/holidays/sargis.html', 'topbicycle/stark.html', 'topbicycle/stels.html']

import glob files = glob.glob("**/[St]*.html", root_dir="/mnt/c/Users/Andrei/glob_ex/", recursive=True, include_hidden=True ) print(files)

['heihei/Spain.html', 'heihei/Sweden.html']

iglob

For working with large amounts of data the method may be useful iglob, which returns an iterator

import glob globs = glob.iglob("**/*.html",             root_dir="/mnt/c/Users/Andrei/glob_ex/",             recursive=True,             include_hidden=True         ) print(globs) print("type(globs): ", type(globs)) print("globs.__next__: ", globs.__next__) print("type(globs.__next__): ", type(globs.__next__)) print(globs.__next__()) print(globs.__next__()) # another way to call next() print(next(globs)) print(next(globs))

<itertools.chain object at 0x7fce30e36ad0> type(globs): <class 'itertools.chain'> globs.__next__: <method-wrapper '__next__' of itertools.chain object at 0x7fce30e36ad0> type(globs.__next__): <class 'method-wrapper'> heihei/Finland.html heihei/Spain.html heihei/Sweden.html heihei/Armenia/.hidden.html

You can go through all files with a for loop

Let us find all .py files in the directory python. For greater clarity of output we use the function enumerate()

import glob globs = glob.iglob("**/*.py",             root_dir="/mnt/c/Users/Andrei/python/",             recursive=True,             include_hidden=True         ) for i, file in enumerate(globs, 2):     print(i, file, sep=': ')

2: copy_ex.py 3: dict_ex.py 4: dir_ex.py 5: pass_arg.py 6: random_gen.py 7: random_gen2.py 8: tcompl_ex.py 9: glob_ex/glob_ex.py 10: glob_ex/glob_ex1.py 11: glob_ex/glob_ex3.py 12: glob_ex/iglob_ex.py 13: Python-3.11.3/python-config.py 14: Python-3.11.3/python.exe-gdb.py 15: Python-3.11.3/setup.py 16: Python-3.11.3/build/lib.linux-x86_64-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py 17: Python-3.11.3/Doc/conf.py … 2934: venv/Lib/site-packages/setuptools/_vendor/tomli/__init__.py 2935: venv/Lib/site-packages/_distutils_hack/override.py 2936: venv/Lib/site-packages/_distutils_hack/__init__.py

Article author: Andrei Olegovich