Two Website Rebuild Mistakes We Made, and How We Fixed Them
An honest account of two technical mistakes from our own 2026 website rebuild, a robots.txt file that blocked crawling and a tangle of conflicting redirects, how we found them, and the checks that prevent them.

Agencies rarely publish what went wrong on their own websites. We think that is a mistake, because the errors we made rebuilding our site in 2026 are the same ones we find on client sites every month. Both were invisible to anyone browsing the site normally, and both could quietly undo months of SEO work.
Mistake 1: a robots.txt file that blocked the whole site
During the build, a physical robots.txt file with a block-all rule was placed on the server. WordPress normally generates robots.txt automatically, and the setting in the dashboard looked correct, but a physical file overrides it. So Google was being told not to crawl the site while everything appeared normal to us and to visitors.
We found it by doing what we tell clients to do: opening /robots.txt directly in a browser, rather than trusting the dashboard. Deleting the file fixed it immediately, and the SEO plugin went back to serving a correct robots.txt with the sitemap listed.
Lesson: after every launch, theme change or server move, open yourdomain/robots.txt and read it. If you see Disallow: / under User-agent: *, your site is blocked.
Mistake 2: conflicting redirects
Over the years, redirect rules had accumulated. Some old URLs redirected to pages that themselves redirected, and a few pointed into each other, creating loops. A loop means the page never loads, and Google records an error. Some published articles were also being redirected away to service pages, so they could never rank.
We exported every rule from the redirection manager, mapped each old URL to its final destination, and checked Search Console to see which old URLs still had search demand. The articles that still had demand were restored as live pages. The rest were pointed directly at the best current page, with no chains, and old URLs showing up in the 404 log were given proper redirects.
Lesson: redirects are not set-and-forget. Audit them after every major change, and never redirect a page that should rank.
The common thread
Neither problem showed up when you clicked around the site. Pages loaded, the design looked right, forms worked. Only three things revealed them: reading robots.txt directly, crawling the site, and looking at Google Search Console data. That is why we treat those three checks as part of every launch, ours and our clients’.
The checks we now run on every launch
- Read robots.txt in a browser on launch day.
- Check the WordPress Reading setting so Discourage search engines is unticked.
- Spot-check noindex tags on the most important pages.
- Crawl the site for redirect chains, loops and broken links.
- Review the redirect list and the 404 log, and make sure no page meant to rank is redirected.
- Submit the sitemap in Search Console and watch the Pages report daily for two weeks.
- Compare Search Console data before and after launch, by page and by query.
If this has happened to you
A blocked robots.txt or a redirect loop can usually be fixed in an afternoon. Recovery in Google then takes days to weeks as pages are recrawled. The sooner you catch it, the less you lose. Start with our website migration SEO checklist and, if you are planning a rebuild, our web design brief guide.
Frequently asked questions
Can a robots.txt file really block a whole website?
Yes. A single line, Disallow: /, tells search engines not to crawl any page. Check yours at yourdomain/robots.txt.
What is a redirect loop?
Two or more redirects that point to each other, so the browser never reaches a page. Google treats it as an error and cannot index the URL.
How often should redirects be reviewed?
After every redesign, migration or URL change, and at least once a year otherwise.



