wget

Introduction

wget is short for the English

web get

Which in Russian sounds like

Get something from the internet

https://www.gnu.org/software/wget/ GNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc. GNU Wget has many features to make retrieving large files or mirroring entire web or FTP sites easy, including: Can resume aborted downloads, using REST and RANGE Can use filename wild cards and recursively mirror directories NLS-based message files for many different languages Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally Runs on most UNIX-like operating systems as well as Microsoft Windows Supports HTTP proxies Supports HTTP cookies Supports persistent HTTP connections Unattended / background operation Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring GNU Wget is distributed under the GNU General Public License.

Download to the current directory

wget http://some_url

An example of downloading CentOS Linux with wget.

To download something useful you need to find where something useful is located

CentOS images are stored in special repositories.

CentOS repositories can be studied on the site centos.org

The list of x86_64 repositories for CentOS 7 can be found

here

For CentOS 8

here

I am in Helsinki so the nearest CentOS repository is ftp.funet.fi

Some other repository from the list may be closer to you

wget http://ftp.funet.fi/pub/mirrors/centos.org/7.9.2009/isos/x86_64/CentOS-7-x86_64-DVD-2009.iso

--2021-11-03 12:36:59-- http://ftp.funet.fi/pub/mirrors/centos.org/7.9.2009/isos/x86_64/CentOS-7-x86_64-DVD-2009.iso Resolving ftp.funet.fi (ftp.funet.fi)... 193.166.3.2, 2001:708:10:8::2 Connecting to ftp.funet.fi (ftp.funet.fi)|193.166.3.2|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4712300544 (4.4G) [application/x-iso9660-image] Saving to: ‘CentOS-7-x86_64-DVD-2009.iso’ CentOS-7-x86_64-DVD 100%[===================>] 4.39G 2.17MB/s in 23m 24s 2021-11-03 13:00:23 (3.20 MB/s) - ‘CentOS-7-x86_64-DVD-2009.iso’ saved [4712300544/4712300544]

As a result, CentOS-7-x86_64-DVD-2009.iso is downloaded to the current directory

Resuming a file

If there is a risk of network interference you can use the flag -c which allows resuming interrupted downloads

wget -c http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso

This flag can be added to others; if there are no interruptions you will not notice a difference.

Specify a directory

If you want to save the image to another directory, use the -P option

In the next command I chose another repository from the list and another image, this time NetInstall

wget -P ~/Downloads http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso

--2021-11-03 13:09:47-- http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso Resolving mirror.hosthink.net (mirror.hosthink.net)... 95.216.21.104 Connecting to mirror.hosthink.net (mirror.hosthink.net)|95.216.21.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 602931200 (575M) [application/octet-stream] Saving to: ‘mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso’

The image CentOS-7-x86_64-NetInstall-2009.iso will be downloaded to the ~/Downloads directory

‘-P prefix’ ‘--directory-prefix=prefix’ Set directory prefix to prefix. The directory prefix is the directory where all other files and subdirectories will be saved to, i.e. the top of the retrieval tree. The default is ‘.’ (the current directory). gnu.org

Also study the option -O

Rename while downloading

If the name is too long for you, it can be changed right away to something more convenient.

wget http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso -O ce7net.iso

The following will be downloaded to the current directory CentOS-7-x86_64-NetInstall-2009.iso and immediately renamed to ce7net.iso

To keep both files use the flag -o

The flag -O will also allow you to specify a directory for saving.

For example:

wget http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso -O ~/Downloads/ce7net.iso

Will save the renamed file to ~/Downloads

GNUmakefile

An example makefile for downloading and renaming the file 0_README.txt with saving to another directory

.PHONY: centos7-readme centos7-readme: # Download centos7_README.txt if [ ! -f ~/Downloads/linux_isos/centos7_README.txt ]; then \ wget -c http://centos.mirror.far.fi/7.9.2009/isos/x86_64/0_README.txt \ -O ~/Downloads/linux_isos/centos7_README.txt; \ else \ echo "centos7_README.txt already exists"; \ fi

The target of this GNUmakefile can be run with the command

make centos7-readme

Read more about make in the article

« Working with GNUmakefile in Linux »

More complex examples of working with wget in a Makefile can be studied in the article

« make + wget »

Keeping the repository hierarchy

If you want to keep the whole directory hierarchy of the repository. For example, if you plan to download several files, use the option -x

wget -x http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso

--2021-11-03 13:09:47-- http://mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso Resolving mirror.hosthink.net (mirror.hosthink.net)... 95.216.21.104 Connecting to mirror.hosthink.net (mirror.hosthink.net)|95.216.21.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 602931200 (575M) [application/octet-stream] Saving to: ‘mirror.hosthink.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso’

A directory hierarchy will be created in the current directory

mirror.hosthink.net/centos/7.9.2009/isos/x86_64/

And the image will be downloaded into it CentOS-7-x86_64-NetInstall-2009.iso

Errors

Warning: wildcards not supported in HTTP.

You cannot just insert * into a url and expect it to work correctly; the server side most likely does not have a normal file system and the search will not work.

Read about how to partially get around this problem in the article

« Wildcard: downloading by partial name »

Article author: Andrei Olegovich