Copying a website

Have you ever wondered how to make a copy of your website? If you're using Linux, you can use WGET to do this. It works on macOS as well but you need HomeBrew.

If you're on macOS, read the next point.

Linux

Open up your favorite terminal tool and type

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://example.org

Each of those parameters means (quoting official documentation):

--mirror

Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing.

--convert-links

After the download is complete, convert the links in the document to make them suitable for local viewing. This affects not only the visible hyperlinks but any part of the document that links to external content, such as embedded images, links to style sheets, hyperlinks to non-HTML content, etc. Because of this, local browsing works reliably: if a linked file was downloaded, the link will refer to its local name; if it was not downloaded, the link will refer to its full Internet address rather than presenting a broken link. The fact that the former links are converted to relative links ensures that you can move the downloaded hierarchy to another directory.

--adjust-extension

If the downloaded resource is of type text/html or application/xhtml+xml, and its URL does not end with ‘.html’ or ‘.htm’, Wget adds the .html extension when saving the file. This ensures that the file will be correctly recognized by the browser, which is important especially when the server generates pages dynamically (e.g., CGI scripts) and the URL ends with, for example, ?id=25, without an extension. Note: files changed in this way may be downloaded again during the next mirroring, because Wget may not recognize that the local file corresponds to the URL — if the URL without the extension never indicated that it was an HTML page.

--page-requisites

It recommends downloading all resources necessary for the correct display of the page, such as style sheets (CSS), images, scripts, fonts — not just the HTML file itself. This makes the local copy of the page look and behave very similarly to the original — with images, styles, etc.

--no-parent

When Wget operates recursively, by default it may also start sorting and downloading “parents” in the directory structure (e.g., higher-level directories). --no-parent blocks this behavior — Wget does not climb up directories. In practice, this means that you only download the content at the address you specified (and the subpages/documents below it), not the entire domain or parent directories.

MacOS

For macOS systems, wget is available in the Homebrew package. Detailed instructions for installing this package can be found here. https://docs.brew.sh/Installation

After completing installation, wget works the same way as in Linux.