wget
Introduction
wget is short for the English
web get
Which in Russian sounds like
Get something from the internet
https://www.gnu.org/software/wget/ GNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc. GNU Wget has many features to make retrieving large files or mirroring entire web or FTP sites easy, including: Can resume aborted downloads, using REST and RANGE Can use filename wild cards and recursively mirror directories NLS-based message files for many different languages Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally Runs on most UNIX-like operating systems as well as Microsoft Windows Supports HTTP proxies Supports HTTP cookies Supports persistent HTTP connections Unattended / background operation Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring GNU Wget is distributed under the GNU General Public License.
Download to the current directory
wget http://some_url
An example of downloading CentOS Linux with wget.
To download something useful you need to find where something useful is located
CentOS images are stored in special repositories.
CentOS repositories can be studied on the site
centos.org
The list of x86_64 repositories for CentOS 7 can be found
here
For CentOS 8
here
I am in
Helsinki
so the nearest CentOS repository is ftp.funet.fi
Some other repository from the list may be closer to you
wget http://ftp.funet.fi/pub/mirrors/centos.org/7.9.2009/isos/x86_64/CentOS-7-x86_64-DVD-2009.iso
--2021-11-03 12:36:59-- http://ftp.funet.fi/pub/mirrors/centos.org/7.9.2009/isos/x86_64/CentOS-7-x86_64-DVD-2009.iso Resolving ftp.funet.fi (ftp.funet.fi)... 193.166.3.2, 2001:708:10:8::2 Connecting to ftp.funet.fi (ftp.funet.fi)|193.166.3.2|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4712300544 (4.4G) [application/x-iso9660-image] Saving to: ‘CentOS-7-x86_64-DVD-2009.iso’ CentOS-7-x86_64-DVD 100%[===================>] 4.39G 2.17MB/s in 23m 24s 2021-11-03 13:00:23 (3.20 MB/s) - ‘CentOS-7-x86_64-DVD-2009.iso’ saved [4712300544/4712300544]
As a result, CentOS-7-x86_64-DVD-2009.iso is downloaded to the current directory
Resuming a file
If there is a risk of network interference you can use the flag -c which allows resuming interrupted downloads
wget -c http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso
This flag can be added to others; if there are no interruptions you will not notice a difference.
Specify a directory
If you want to save the image to another directory, use the -P option
In the next command I chose another repository from the list and another image, this time NetInstall
wget -P ~/Downloads http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso
--2021-11-03 13:09:47-- http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso Resolving mirror.hosthink.net (mirror.hosthink.net)... 95.216.21.104 Connecting to mirror.hosthink.net (mirror.hosthink.net)|95.216.21.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 602931200 (575M) [application/octet-stream] Saving to: ‘mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso’
The image CentOS-7-x86_64-NetInstall-2009.iso will be downloaded to the ~/Downloads directory
‘-P prefix’ ‘--directory-prefix=prefix’ Set directory prefix to prefix. The directory prefix is the directory where all other files and subdirectories will be saved to, i.e. the top of the retrieval tree. The default is ‘.’ (the current directory). gnu.org
Also study the option -O
Rename while downloading
If the name is too long for you, it can be changed right away to something more convenient.
wget http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso -O ce7net.iso
The following will be downloaded to the current directory
CentOS-7-x86_64-NetInstall-2009.iso
and immediately renamed to
ce7net.iso
To keep both files use the flag -o
The flag -O will also allow you to specify a directory for saving.
For example:
wget http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso -O ~/Downloads/ce7net.iso
Will save the renamed file to ~/Downloads
GNUmakefile
An example makefile for downloading and renaming the file 0_README.txt with saving to another directory
.PHONY: centos7-readme
centos7-readme:
# Download centos7_README.txt
if [ ! -f ~/Downloads/linux_isos/centos7_README.txt ]; then \
wget -c http://centos.mirror.far.fi/7.9.2009/isos/x86_64/0_README.txt \
-O ~/Downloads/linux_isos/centos7_README.txt; \
else \
echo "centos7_README.txt already exists"; \
fi
The target of this GNUmakefile can be run with the command
make centos7-readme
Read more about
make
in the article
« Working with GNUmakefile in Linux »
More complex examples of working with wget in a Makefile can be studied in the article
« make + wget »
Keeping the repository hierarchy
If you want to keep the whole directory hierarchy of the repository. For example, if you plan to download several files, use the option -x
wget -x http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso
--2021-11-03 13:09:47-- http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso Resolving mirror.hosthink.net (mirror.hosthink.net)... 95.216.21.104 Connecting to mirror.hosthink.net (mirror.hosthink.net)|95.216.21.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 602931200 (575M) [application/octet-stream] Saving to: ‘mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso’
A directory hierarchy will be created in the current directory
mirror.hosthink.net/centos/7.9.2009/isos/x86_64/
And the image will be downloaded into it CentOS-7-x86_64-NetInstall-2009.iso
Errors
Warning: wildcards not supported in HTTP.
You cannot just insert * into a url and expect it to work correctly; the server side most likely does not have
a normal file system and the search will not work.
Read about how to partially get around this problem in the article
« Wildcard: downloading by partial name »
Article author: Andrei Olegovich