How Wayback Machine restoration works: a technical guide

Restoring a website from the Wayback Machine is a data problem before it is a design problem. You are reconstructing a site from an index of individual captures, each one a response recorded for one URL at one moment, and the quality of the rebuild depends on choosing the right ones. This guide is for whoever will do that work: a developer or technical SEO who is comfortable with HTTP and some scripting.

It covers what a capture is, where captures come from, how to query the Internet Archive's CDX index, the traps in the data, and a pipeline that turns it into a clean site. One rule sits above all of it: archive presence is not permission. The Internet Archive says it "does not make guarantees as to the copyright status of items", and who may reuse recovered content is covered, with the non-technical steps, in how to restore a website from the Wayback Machine.

A capture is a stored copy of the response a crawler received for one URL at one time. It is identified by the URL and a timestamp: the number in the middle of an archive address, which the Internet Archive's help pages say "translates as yyyymmddhhmmss". Fourteen digits, to the second.

Three properties matter for restoration:

• Each capture keeps its status. The calendar colours a dot blue for a 2xx response, green for 3xx, orange for 4xx and red for 5xx. A redirect or an error is recorded as faithfully as a working page.
• Every file is captured on its own. A page, each of its images and each stylesheet have separate captures with separate timestamps, which is why one archived page can be assembled from several dates.
• Captures are not immediate. The help pages describe "a 3-10 hour lag time between the time a site is crawled and when it appears in the Wayback Machine."

Capture density varies enormously between sites, and between pages on the same site, because captures arrive by several routes. The Internet Archive says "hundreds of web crawls" contribute every day. Individuals add pages with Save Page Now, which saves "the page you enter including the images and CSS" but "does not save any of the outlinks"; it saves one page, not a site, and does not add the URL to future crawls. Archive Team, a volunteer group, saves sites that often end up in the Wayback Machine, and Archive-It is a subscription service through which organisations run their own crawls.

The upshot: a homepage that people kept saving by hand can have hundreds of versions while the inner pages of the same site have one or none. Plan a restore around what is actually there, which means building an inventory before anything else.

The CDX server is the index the Wayback Machine itself uses. Its README describes "a standalone HTTP servlet that serves the index that the wayback machine uses to lookup captures". It answers GET requests at web.archive.org/cdx/search/cdx and returns plain text or, optionally, a JSON array.

Two caveats before you build on it. The README is titled "Wayback CDX Server API - BETA", and its last change in the GitHub repository was in October 2018, so treat the parameter descriptions as indicative and test behaviour against the live service. When we checked on 7 October 2026 it answered in the documented format. A query for the latest capture of example.com returned this single line, with the seven public fields in order:

com,example)/ 20261007031409 https://example.com/ text/html 200 U2352G3H6BFNEDPBNQI5XGOBSQAN4W4J 1093

The parameters that matter

• matchType widens a query beyond the single URL you give. matchType=prefix returns "all results under the path", and matchType=domain returns the host "and all subhosts". A prefix query on the host covers the whole site; domain adds subdomains.
• from and to restrict results to a timestamp range.
• filter takes the form filter=field:regex, keeps matching rows, and can be repeated; an exclamation mark before the field name inverts it. filter=statuscode:200 together with filter=mimetype:text/html leaves working HTML pages only.
• collapse takes collapse=field, or collapse=field:N to compare only the first N characters, and drops rows that duplicate the row before them. collapse=digest removes consecutive identical versions. Because the index is usually sorted by URL and then date, collapse=urlkey normally leaves one row per URL. The README is explicit that "only adjacent digest are collapsed, duplicates elsewhere in the cdx are not affected": a page that changes and later changes back is counted again.
• limit matters because, in the README's words, the server "may return millions or billions of record", and its configuration caps a single query (150,000 rows by default, per the README). Split large sites by path or date range, and write code that expects a truncated answer.

Suppose you are restoring example-domain.test, a small business site that ran from about 2013 until its domain lapsed. A first query lists every working HTML page once:

https://web.archive.org/cdx/search/cdx?url=example-domain.test/&matchType=prefix&filter=statuscode:200&filter=mimetype:text/html&collapse=urlkey

A second asks for the full history of the site at every status, with consecutive identical captures folded together:

https://web.archive.org/cdx/search/cdx?url=example-domain.test/&matchType=prefix&collapse=digest

Merged and trimmed to the interesting rows, an illustrative answer might look like this:

Reading it as a restorer would:

• Three designs, then something else. The homepage has distinct digests in 2013, 2017 and 2019, which look like real versions of the site. The 2024 row has a new digest and a fraction of the length. On a lapsed domain that pattern is worth opening, because a holding or parking page recorded after the business stopped would look exactly like this. Confirm by looking at the capture, not by the number.
• A URL change the old site made itself. /index.php?page=services in 2016 and /services/ in 2019 are the same page under two URL schemes. Other sites may link to either, so both belong in the redirect map.
• A redirect kept as evidence. The 301 on /offers/ tells you the old site redirected that address in 2020. The seven public fields do not include the redirect's target, so to learn where it pointed you have to open the capture.
• Protocol drift. The move from http to https in 2019 means the same homepage appears under two original URLs. Normalise scheme (and www, where the site used it) before counting pages, or you will count the site twice.

The simpler Wayback Availability API (archive.org/wayback/available?url= followed by the address) answers one question: is there a capture near a given time? It returns "a single closest snapshot", takes an optional timestamp of "1-14 digits (YYYYMMDDhhmmss)", and returns the most recent capture if you give none. Its documentation, last updated in 2013, suggests it for 404 handlers that look for an archived copy to display.

When nothing is available the response is {"archived_snapshots":{}}, which the documentation defines as "not archived or currently not accessible". Those are two different situations, and the response does not say which applies. In our own test on 7 October 2026, the Availability API gave that empty answer for example.com and for archive.org/about/terms.php, while the CDX API listed captures of both; a request for bbc.co.uk returned a snapshot normally. That is one day's observation by DropOps, not a statement by the Internet Archive, but it is enough to set a rule: never conclude that a site was not archived from an empty Availability response. Ask the CDX index.

When you browse an archived page you are not looking at a single moment. For links missing at the date you are viewing, the help pages say the Wayback Machine "will grab the closest available date", and where it has no copy at all it "will look for the link on the live web and grab it if available". The rendered page can therefore combine a 2015 HTML capture, a 2017 stylesheet, a 2012 logo and an image served today by whoever now controls that address.

That is helpful for browsing and hazardous for restoration. A rebuild made by saving what the browser shows inherits the blend without recording it. Work from the index instead: for each page you keep, record its timestamp and the timestamp of every asset taken with it, and set a tolerance (a few months either side, say) beyond which an asset gets a second look. A stylesheet captured three years after the HTML may belong to a redesign that page never had.

Recovered HTML carries the old site's links in several forms, and each needs a decision during the rebuild:

• Relative links such as ../services/ or contact.php resolve against the page's original URL. Resolve them against the original field from CDX, not the archive address you fetched from, then map the result to the new site.
• Absolute links to the old site appear in every variant the site ever used: http and https, with and without www, with and without index.php. Normalise them to one form, then map.
• Links to other sites need checking individually. Pew Research Center found in 2024 that 38% of web pages that existed in 2013 were no longer accessible a decade later.
• Links into the archive. Anything saved from the playback view, rather than taken as content, may contain archive addresses: web.archive.org/web/ followed by a timestamp and the original URL. Rewrite every one. A restored site must not depend on the archive at run time.

Status codes need the same care. A 3xx capture is evidence that the old site had moved an address itself, and it belongs in your redirect map. A 4xx or 5xx capture records a page that was failing at that moment, not necessarily for good. And a 200 is not proof of content: a parked domain, a "for sale" notice and a hacked page full of spam links can all return 200. Read the status code as what the server said, then look at what it served.

Some gaps are structural. The archive collects "web pages that are publicly available"; it does not archive pages that need a password, or pages "only accessible when a person types into and sends a form". Orphan pages are missed ("If there are no links to your pages, the robot won't find it"), crawlers "don't enter queries in search boxes", and JavaScript that generates links "without having the full name in the page" hides those links from the crawler. Site search results, filtered product lists and member areas are the usual casualties.

Other gaps come from exclusion. Sites "may have been excluded from the Wayback Machine due to a robots.txt file on the site or at a site owner's direct request", and the Internet Archive reviews removal requests without making "any guarantees beforehand about the outcome". Robots.txt has a history worth knowing for dropped domains. In 2017 the Internet Archive wrote that parked domains blocking search engines through robots.txt had "historically also removed the entire domain from view in the Wayback Machine". Having stopped referring to robots.txt files on US government and military sites, it said it was "looking to do this more broadly". That was a stated intention, not a published final policy, and current help pages still say robots.txt "may have prevented the crawling of a site". The practical consequence: what is visible for a domain can change, so record what you found and when.

For a sense of scale, DropOps measured the homepage capture history of a sample of dropped .uk domains. For each domain we made one CDX request for the bare domain with collapse=digest and counted HTTP 200 captures, so the figure is distinct homepage versions, not visits; and because collapsing only folds adjacent duplicates, a homepage that changes and changes back counts again. Across 7,310 domains with homepage history, dropped between 16 August and 30 September 2026, the median was 33 versions, with an interquartile range of 14 to 62. The sample was weighted towards domains with strong link evidence, so it describes that sample, not every dropped name, and it says nothing about inner pages, images or files. The full study, with its method and limits, is in what can actually be recovered from an archived website.

The CDX README gives no rate-limit figures, so this guide does not offer any. The Internet Archive's terms describe access to its collections as provided "at no cost to you", which is reason enough to behave like a considerate guest:

• Cache every response locally, keyed by URL and timestamp, and never fetch the same capture twice. A digest you already hold is content you already have.
• Ask the index for a whole prefix in one query rather than making one query per URL.
• Fetch sequentially with a pause between requests, not in bursts of parallel connections.
• Back off when responses slow down or errors appear, and stop rather than retry in a tight loop.
• Identify your tool with a descriptive user agent and a contact address.

Step seven is where the archive's evidence turns into lasting value, and it is a job of its own. How to recover old URLs and build a redirect map turns the inventory from step two into a prioritised, one-hop map, and how to rebuild an old website without losing its SEO covers the launch and the checks afterwards.

DropOps Sites, our website builder for domains you own, follows the same pipeline in outline. It reads a domain's old addresses and a sample of old pages from public web archives, strips scripts, trackers and embeds from anything it recovers, hotlinks nothing from the archive, and holds old details such as phone numbers and prices until the owner confirms them. Nothing goes live until the owner publishes. What an archive holds for any given domain varies, and none of it is guaranteed.