How To Download Absolutely Everything You Can Find On The Web

Image: Gizmodo

The internet is always there, vast and accessible, from wherever you happen to be — until it isn’t. Maybe you’ve got a long flight ahead of you and need some reading material, maybe you just want to store something for safe-keeping. Whatever your reason here’s how you can download just about anything you come across on the internet with free tricks and tools.

To quickly save a web page

Your web browser can actually save webpages and other elements on its own, though you might not have ever realised it. In Chrome, right-click on any page and choose Save as to commit it to disk — make sure Web Page, Complete is the save type. You get all the text and most of the images, but no embedded media like videos.

The process is similar in Firefox. Again just right-click on a blank area of the page you’re on, choose Save Page As from the menu that pops up, and give your page a name and location (making sure Web Page, complete is selected as the save type). Aside from any streaming videos, audio, or complicated interactive elements, you should have a pretty good facsimile of the page on your local system.

In Safari, open a page then choose File and Save As. Give your new file a name, make sure the Web Archive format is selected, and you’ve got pretty much a carbon copy of the site you were on, adverts and JavaScript and all — when you open it up again you’ll be hard pushed to tell that you’re working offline, though again embedded media gets missed out.

These save tools are convenient but not very comprehensive — they were built in a simpler time when webpages were static and straightforward blocks of text and pictures thrown up in your web browser. The option is handy to have at a push, but you may well notice broken or missing elements using this method.

To save a web page AND all the good stuff on it 

Read-it-later services like Pocket and Instapaper are fine for getting webpages into a simplified form that you can come back to at a later date, but for those times when you want to suck down everything on a page in its entirety you need something more than the integrated tools you get in your browser.

One of the longest-serving and best-known tools for the job is HTTrack for Windows, which more or less creates a complete cache of a site on your local disk for you — it’s just like viewing the actual site, only everything is offline, including pictures and fonts and links to pages on the same domain. Your downloads can be categorized and titled, and cover multiple URLs inside the same project.

Another option is Cyotek WebCopy, again for Windows. It’s very similar in operation, crawling around a site and downloading absolutely everything on it, including images and videos (as long as they’re hosted on the site and not being streamed from somewhere else). It’s even more comprehensive and user-friendly than HTTrack, but they’re both well worth exploring for copying entire sites and everything on them.

If you’re more of a macOS person, give SiteSucker a go. It will set you back $US5, but getting content cached locally is as easy as entering a URL and then watching the program go to work — it grabs text, images, stylesheets and more, with everything wrapped up in its own localised folder. It can even pause and resume downloads.

Images

The applications we’ve just mentioned should pull down everything from a particular site, including images and videos hosted on the site itself, but there might be times when you just want to grab a picture or two without sifting through a whole HTML page and all its elements — the reams of folders you can potentially end up with aren’t the easiest to sift through if you’re just after a couple of images.

Of course there’s the usual right-click-and-save option available as part of your browser, but this isn’t always available: You could be trying to get an image that’s hidden away in the background or embedded as part of the stylesheet, or a picture that can’t be saved because it’s covered by a link.

Gallerify for Chrome is one of the best tools we’ve come across for this. Click the extension icon and you get a list of all the images embedded on a page, complete with original resolutions. One click downloads them all, or you can open them up in separate tabs and download them individually.

For something even simpler, try Download All Images for Chrome. Open up a page, click the extension icon, and everything is packed together and saved to disk in a zip file. Unpack the archive and you’ve got copies of every image on the site waiting for you — though we found it’s not quite as good as Gallerify at picking up hidden or background images not displayed in the usual way.

For those Firefox fans among you, we can recommend Download All Images for its ease-of-use and also its customizability. Click the extension icon and a dialog pops up, letting you filter images by size, dimensions, or type — you can pull down just the pictures you need from the page you’re on.

For Safari, there isn’t really an equivalent image downloader you can take advantage of, though Save Images will do the job on an iOS device. For simplicity’s sake, your easiest option on a Mac is to install Chrome or Firefox for the purposes of downloading all the images from a site, and use the add-ons those browsers have available.

Videos and audio

What these site grabbers won’t get, a lot of the time, is embedded video and audio that’s designed to be viewed online and not saved to disk (like YouTube and Spotify streams). A lot of this material is protected by copyright — so you run the risk of falling foul of your country’s laws if you rip this kind of material. Please proceed at your own risk.

If you absolutely must get the video and audio content, then again browser extensions are your friend. Chrome Audio Capture, as the name suggests, will capture audio inside any Chrome tab — it’s not particularly sophisticated, but then it doesn’t have to be. You simply choose your format then start and stop the recording as needed.

Over on the Firefox side of the fence, Audio Downloader Prime is a more sophisticated extension that can identify individual audio elements on a page and let you download them individually. It did seem a little hit and miss when we tested it, but if you need to download audio in Firefox, it’s worth a try.

For video downloads, desktop software is often your best bet: 4K Video Downloader works across some of the most popular video streaming sites and services, is available for Windows and macOS, and won’t cost you anything to use (unless you need to download a lot of video or want to include extras like captions).

There are a wealth of other options out there, plenty of which we’ve tested, but a lot of them are unreliable or scammy: They will try and install something extra with the main program, or put a watermark on your clips, or change your browser’s homepage, or ask for personal details before getting on with the download.

Remember that the likes of YouTube are very keen to stamp down on these illicit tools, and malware makers are very keen to make some cash out of them, so be wary about which software you use for the job. We haven’t found any apps that can touch the three we’ve already listed — even KeepVid has now gone legit.