Downloading an entire website or part of it

A useful method for getting a website you need within code or from the command line.

wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains \
     --no-parent \

This wget command downloads the website in it’s entirety following hyperlinks recursively.

What it all means:

  • –recursive
    • download the entire website.
  • –domains
    • don’t follow links outside
  • –no-parent
    • don’t follow links above or outside the directory examples/.
  • –page-requisites
    • get all the elements that compose the page (images, CSS and so on).
  • –html-extension
    • save files with the .html extension.
  • –convert-links
    • convert links so that they work locally, off-line.
  • –restrict-file-names=windows
    • modify filenames so that they will work in Windows as well.
  • –no-clobber
    • don’t overwrite any existing files (used in case the download is interrupted and resumed).

Wget is available for most operating systems, pre-installed in most versions of Linux and can be downloaded as a binary for windows too.