Sitemap and Robots.txt: Small Files, Expensive Mistakes
Two files decide what Google is allowed to look at and what it knows exists. One wrong line can hide an entire site.
Why blocking a page keeps it in Google
Blocking stops Google crawling. If the page is already indexed it stays, and Google can no longer read the tag that would remove it.
-
The files get set up
Usually by a plugin, sometimes by a developer working from a template built for a different site.
-
Something quietly breaks
A leftover disallow rule, a sitemap that failed to parse, or entries pointing at pages deleted a year ago.
-
They get corrected
Sitemap rebuilt to list only live indexable pages, blocks replaced with the right instruction, both verified with Google tools.
Two small files decide what Google is allowed to look at and what it knows exists.
Get them wrong and you can hide an entire site by accident. Get them right and they quietly do their job forever.
They are also the two files most often broken by a plugin update or a developer working from an old template.
The Honest Version: These Files Do Less Than People Think
A sitemap does not get you indexed
It tells Google which pages exist. It does not oblige Google to keep any of them. Submitting a sitemap for pages Google has already looked at and declined changes nothing at all.
People resubmit sitemaps repeatedly hoping it forces indexing. It has never worked that way.
Robots.txt does not hide pages
This one causes real damage. Blocking a page stops Google crawling it, but if the page is already indexed it stays in the index, now with Google unable to read it.
Worse, a blocked page means Google cannot see a noindex tag on it. People block pages in order to remove them and achieve the exact opposite of what they wanted.
Where they genuinely matter
Sitemaps matter on large sites and new sites, where discovery is a real problem. Robots.txt matters for keeping crawlers out of areas that generate endless addresses, like filters and search results. Outside those two situations, they are hygiene rather than strategy.
Block, Noindex or Remove: Which One You Actually Want
These three get confused constantly, and picking the wrong one is why pages refuse to disappear.
Use noindex when you want it out of Google
The page stays live and reachable, and Google is told not to keep it. This is the correct choice for thin pages, thank-you pages, internal search results and anything you do not want found but do want working.
Google must be able to crawl the page to see this instruction. That is the part people get wrong.
Use robots.txt when you want to save crawling
Correct for areas that generate infinite addresses, where you do not care what happens to them and you never want Google spending time there. It is about crawl efficiency, not about removal.
Use removal requests when you need it gone today
Search Console has a removal tool that hides a page from results within hours. It is temporary, lasting about six months, so it buys you time while a proper noindex takes effect. Useful for something published by mistake.
The order that works
If a page must go immediately: request removal, add noindex, leave it crawlable. Never block it in robots.txt while you are trying to remove it, because that is what stops the removal working.
What Actually Goes Wrong
Small files, expensive mistakes.
Disallow: /
One line that blocks the entire site. Usually left over from development. The single most costly typo in SEO.
Blocking pages to deindex them
A common misunderstanding. Blocked pages stay in the index and Google can no longer read the tag that would remove them.
Sitemaps full of dead URLs
Listing redirects, 404s and noindexed pages. Wastes crawl and signals that nobody maintains the site.
A sitemap that was never read
Submitted once, failed to parse, and nobody checked. Search Console reports this clearly and almost nobody looks.
Several conflicting sitemaps
An old plugin and a new one both generating one, listing different pages, with the stale one still submitted.
Blocking CSS and JavaScript
Prevents Google rendering the page properly, so it judges a version of your site that no visitor ever sees.
What You Can Check Yourself First
Both files are public. You can read your own in under two minutes.
1. Open yourdomain.com/robots.txt
Read every line. You are looking for Disallow: / on its own, which blocks everything. Also look for rules covering folders you did not intend to block, such as an entire blog directory.
2. Open your sitemap
Usually at /sitemap.xml or /sitemap_index.xml. Count roughly how many addresses it lists and compare that with how many pages you actually have. If it lists far more, it is including things that should not be there.
3. Click five sitemap URLs at random
Every one should load a real page and return normally. If any redirect or come back not-found, your sitemap is feeding Google addresses that no longer work.
4. Check the Sitemaps report in Search Console
It shows the last read date and whether it succeeded. A sitemap that has not been read in months, or that shows an error, is doing nothing for you.
What I Do
1. Read what is actually there
Including rules added by plugins nobody remembers installing, and rules inherited from a template written for a completely different kind of site.
2. Rebuild the sitemap to list only live, indexable pages
No redirects, no noindexed pages, no 404s, no duplicates. A sitemap is a statement of what you want indexed and it should be accurate.
3. Replace blocking with the correct instruction
Where pages are blocked in order to remove them, the block comes off and a noindex goes on, so Google can read the instruction and act on it.
4. Verify with Google’s own tools
The robots tester and URL inspection confirm what Google can and cannot reach. Not what the plugin claims. What Google reports.
When You Do Not Need This
Your pages are getting indexed fine. Then both files are doing their job. Leave them alone.
You have a small site. Under a few hundred pages, Google will find everything through your internal links regardless of your sitemap.
A tool flagged a warning. Most sitemap and robots warnings are informational. I will tell you which of yours actually matter, usually for nothing.
Price and Turnaround
Scope |
Price |
Turnaround |
|---|---|---|
Audit and repair |
$149 |
1 – 3 days |
Rebuild for a large or ecommerce site |
$349 |
4 – 7 days |
Complex, multi-sitemap or multi-domain |
From $549 |
1 – 2 weeks |
Small job, and often the cheapest thing here that produces a visible change, because a blocked site starts appearing again quickly once the block is removed.
Common Questions
Do I need a sitemap at all?
For a small, well linked site, not really. For large or new sites it genuinely helps discovery.
How do I remove a page from Google properly?
Add a noindex tag and let Google crawl it. Do not block it in robots.txt, because then Google cannot see the tag.
Should my sitemap include every page?
Only pages you want indexed. Including noindexed or redirected pages sends contradictory signals.
How often should the sitemap update?
Automatically, whenever you publish. Most plugins handle this. The usual problem is stale entries rather than missing ones.
Can I block scrapers with robots.txt?
No. It is a request, and only well behaved crawlers honour it. Anyone scraping you will ignore it completely.
My sitemap shows an error in Search Console.
Then it is not being read, which means it is doing nothing. Worth fixing and usually quick.
Is there any harm in a large sitemap?
Not in size itself. The harm is in accuracy. A large sitemap of live pages is fine; a small one full of dead links is not.
Open your robots.txt right now
If you see a line reading Disallow followed by a single slash, send me a screenshot. That one is urgent and free to confirm.
Or email info@shazzseo.com