Downloading an entire website or part of it

A useful method for getting a website you need within code or from the command line.

wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains example.com \
     --no-parent \
         http://www.example.com/examples/

This wget command downloads the website www.example.com/examples/ in it’s entirety following hyperlinks recursively.

What it all means:

* --recursive
  * download the entire website.
* --domains example.com
  * don't follow links outside example.com.
* --no-parent
  * don't follow links above or outside the directory examples/.
* --page-requisites
  * get all the elements that compose the page (images, CSS and so on).
* --html-extension
  * save files with the .html extension.
* --convert-links
  * convert links so that they work locally, off-line.
* --restrict-file-names=windows
  * modify filenames so that they will work in Windows as well.
* --no-clobber
  * don't overwrite any existing files (used in case the download is interrupted and resumed).

Wget is available for most operating systems, pre-installed in most versions of Linux and can be downloaded as a binary for windows too.

Leave a Comment