Wget is a free utility for download of files from the web. It is non-interactive, so can work in the background.

Download a file from GitHub or Gisthub

Click “raw” to access the URL of the raw file.

To download into the current directory:

wget https://link-to-raw-file.php

Download a List of Pages

This can be a useful way to collect images from a service like Unsplash. Image download links can be added on a separate line in a manifest file, which can be used by wget:

wget -i manifest

Download All Images From a webpage

wget -nd -H -p -A jpg,jpeg,png,gif -e robots=off http://www.website.com

• -nd or --no-directories When retriving recursively, wget will not create a hierarchy of directories - all files will be saved to the current directory.
• -H Allow spanning across hosts when doing recursive retrieving - be careful since foreign hosts may link to other hosts, sucking down more data than intended.
• -p Page requisites - allows wget to downlaod all files necessary to display a HTML page, including inline images, sounds and stylesheets.
• -A Specify a comma separated list of files to accept - in this example, image files.
• -e Execute command as if it were part of the file .wgetrc. Executed after .wgetrc, therefore taking precedence - needed for the robots=off flag.

• robots=off Turns off robots exclusion - by default wget respects robot exclusion rules set in the site /robots.txt

• wget syntax guide
• wget man page