How to recover old URLs and build a redirect map

A redirect map has two columns: the addresses the old site used, and where each one should go now. The right-hand column is a matter of judgement. The left-hand column is where most redirect projects quietly fail, because nobody holds a complete list of old URLs, least of all after a site has been offline for a year or has changed hands.

This guide is about building that list from evidence: where old URLs can come from, what each source gets wrong, how to clean and rank them, and how to turn them into rules that send each one, in a single hop, to a page that does the same job. It is the detailed version of the inventory and mapping steps in how to rebuild an old website without losing its SEO, which covers the rest of a launch.

No single source is complete. Google's site-move guide suggests looking in your sitemaps, server logs or analytics, the Links report in Search Console, and your content management system. When the site has gone, public web archives and backlink data join the list, and often lead it.

Querying the Wayback Machine's CDX index

For a site that has gone, the archive's CDX index is usually the first thing to query, because it lists the addresses the Wayback Machine captured, each with a capture date, a file type and the HTTP status the crawler received. Two requests do most of the inventory work. The first lists every captured address on the domain and its subdomains, roughly one line per address:

https://web.archive.org/cdx/search/cdx?url=example-domain.test&matchType=domain&collapse=urlkey

The second finds the addresses the old site was already redirecting. Some may still carry links from an earlier version of the site, so they belong in your map too:

https://web.archive.org/cdx/search/cdx?url=example-domain.test&matchType=domain&filter=statuscode:3..&collapse=urlkey

From there, add filter=statuscode:200 and filter=mimetype:text/html to see which addresses were ever captured as working pages, and from= and to= to split a large site by year, because a single query is capped. What each parameter does, the seven fields on every line and how to query the index without overloading it are covered in how Wayback Machine restoration works. For a quick look before you script anything, the Internet Archive's help pages point to a browser view of the form web.archive.org/*/www.yoursite.com/* that lists every archived file for a site.

Two cautions matter for an inventory. The archive is a sample: the Internet Archive says some sites are missed because its crawlers "were unaware of their existence at the time of the crawl", and that "If there are no links to your pages, the robot won't find it". And it records whatever happened at an address after the real site ended, which the cleaning step below deals with.

Old sitemaps in the archive

Ask the CDX index for /sitemap.xml, /sitemap_index.xml and any sitemap address named in an archived robots.txt. If the archive captured them, they are the old site's own statement of what it published. Captures from different years show how the structure changed, which helps when one old URL pattern replaced another.

Backlinks: the URLs that matter most

A backlink export lists the old URLs that other sites link to and how many separate sites link to each. For redirect planning it is the most valuable source on the list, because an inbound link keeps bringing visitors whether or not anyone remembers the old site. It is still third-party crawl data, so treat the counts as estimates, expect tools to disagree, and discount obviously spammy links. Rank by the linking sites you can actually see rather than a tool's score: Google has said that third-party "reputation" or "authority" scores "don't correspond to any of Google's own signals".

Raw exports from these sources list the same page many times over. Normalise every URL to one form before you count, rank or map anything, but keep the original variants alongside: the redirect rules have to catch the forms that people actually link to.

Then set aside what is not a page. Keep images, PDFs and other downloads on a list of their own rather than deleting them: Google's site-move guide notes that images and downloads "might already be getting traffic from Google Search or links". Old theme files, scripts and stylesheets rarely need redirects once no page uses them. Finally, flag anything from the archive that post-dates the real site, such as parking pages, registrar holding pages or redirects to unrelated sites, and any pattern that looks injected, such as hundreds of pharmacy or casino paths in a site's final months. Those are not part of the site you are rebuilding, though they still need a decision.

The destination to avoid is the default. A rule that sends every unmatched URL to the homepage is easy to write and unhelpful to everyone who hits it: a visitor who followed a link to a specific article gets a generic page and has to start again. Google's site-move guide says that redirecting many old URLs "to one irrelevant single URL destination, such as the home page of the new site" can "confuse users and might be treated as a soft 404 error". A clear not-found page with search and clear navigation serves that visitor better than a redirect that pretends to answer the question.

Choosing between 404 and 410 is a matter of accuracy, not speed. RFC 9110 prefers 410 when the server knows the condition "is likely to be permanent"; Google says 4xx codes other than 429 "are treated the same". Pick whichever is true.

A small site can be mapped by hand in an afternoon. A site with thousands of recovered addresses needs an order of work, and that order should follow evidence of value, not how the URLs look. The useful evidence, roughly in order of weight:

1. Separate sites linking to the URL. The number of distinct linking domains says more than the raw link count, which a single sitewide footer can inflate.
2. Historic search and visitor traffic, if you have analytics or Search Console exports.
3. Archive evidence of a real page: captures with a 200 status, several distinct versions over the years (the digest field changes when the captured content changes), and a last working capture near the site's final period.
4. Content type. Substantial guides, service and product pages and downloadable files usually matter more than tag archives, paginated lists and internal search results.

That evidence sorts addresses into tiers. The thresholds are yours to set from the shape of the data; the table shows one way to do it.

A small garden-design business ran its site on example-domain.test from about 2012 to 2022. Its hosting account was then closed and the backups went with it, though the business kept the domain. It is now rebuilding on a new platform with clean addresses. There are no analytics, no Search Console exports and no server logs. The inventory came from a CDX query, an archived 2019 sitemap and a backlink export. After normalising, eleven addresses had enough evidence to deserve an individual decision.

Three decisions in that map are easy to get wrong. The two lawn-care posts are a genuine merge: one new guide replaces both, so both redirect to it, which Google's site-move guide allows for content consolidated onto a single page. The portfolio address carries its meaning in the query string, so a rule matching only the path /portfolio.php would send every project to the same place; the rule has to read id=17. And /shop/ is not forced into an answer. It has links, but the business no longer sells seed kits, so whether to build a new page or retire the address is the owner's decision, not the spreadsheet's.

The spam path is recorded and retired rather than ignored. If it attracted links while the site was compromised, requests for it will keep arriving, and a 410 tells every client that the content is intentionally gone.

How you implement the map depends on where the site runs. The rules below implement part of the example. They are illustrative: test them on a staging copy, and check your server's or host's documentation for the exact syntax it supports. Put specific rules before general ones, and handle protocol and host in the same hop as the path, so that nobody arriving on http://www. needs two redirects to reach the right page.

Apache (.htaccess with mod_rewrite)

The path rules come first and use absolute destinations, so a request on any protocol or host lands in one hop. The catch-all host and protocol rule goes last. Behind a CDN or proxy, check how your host reports HTTPS before relying on the %{HTTPS} test.

nginx

Exact-match location blocks suit a handful of addresses; a map is easier to maintain for a long list. The plain-http and www server blocks should apply the same rules with absolute destinations, rather than bouncing every request to the chosen host first.

A _redirects file

Some static hosts read a plain-text _redirects file with one rule per line: the old path, the new path and the status code. Retired pages need no line, because a page that does not exist gets the host's not-found response; check that it returns a real 404 status. Support for query-string matching and for status codes other than redirects varies between hosts that read this format, so check your host's documentation before relying on either.

Run the whole old-URL list, variants included, against a staging copy before launch and against the live site afterwards. For each URL, record every hop and its status code; a crawler in list mode does this, and so does a short script around a command-line HTTP client. A pass looks like this:

• Redirected URLs make exactly one hop, return 301 or 308, and land on the mapped URL, which returns 200.
• Kept URLs return 200 at the old address.
• Retired URLs return 404 or 410, never a 200 page that says "not found", which Google reports as a soft 404.
• There are no loops, and no chain longer than one hop without a reason you can name. Google's crawler "generally follows 10 redirect hops", but its site-move guide advises redirecting "to the final destination directly".
• The http, www, trailing-slash and mixed-case forms of your Tier 1 URLs also arrive in one hop.

After launch, the work becomes monitoring. In Search Console's Page indexing report, "Redirect error" covers chains that are too long, loops and bad URLs in a chain, and unexpected "Not found (404)" entries are usually old URLs missing from the map. Server logs are the best source of addresses nobody found during the inventory: filter for 404s on old-looking paths, give each one a decision and add it to the map. Keep the map as a maintained document next to the rules, and keep the redirects in place. Google's site-move guide says "generally at least 1 year", and from users' perspective suggests "keeping redirects indefinitely".

DropOps Sites, our website builder for domains you own, follows this method when it rebuilds on a domain with history. It reads the old addresses and a sample of old pages from public web archives such as the Internet Archive's Wayback Machine, then gives each important address one of the decisions above, or a clean new address reached by a single 301. The decisions rest on evidence such as linking sites and archive records, not on the AI. It does not create chains or loops or send old pages to the homepage, and an address whose evidence could not be gathered goes to human review rather than being retired silently. The HTML export carries the finished map as .htaccess 301s and a _redirects file, alongside a sitemap and a 404 page.

Addresses are the structural half of bringing a site back. For the content half, see how to restore a website from the Wayback Machine.