How to fix robots.txt and your sitemap

Two files, two different jobs

robots.txt is a short text file at the root of your site that tells search engine crawlers which addresses they may fetch. The sitemap is a list of the pages you want search engines to know about. One says “keep out of here”, the other says “here is what I have”. They work together, and most problems come from the two disagreeing.

It helps to keep two more page-level signals in mind, because people often reach for robots.txt when they mean one of these. A noindex tag asks Google not to list a page. A canonical tag tells Google which copy of a page is the main one.

How to read your robots.txt

Open https://yourdomain.com/robots.txt in a browser. The file must sit at the root of the host, and each host has its own: shop.example.com needs its own file. A short, healthy file looks like this:

User-agent: *
Disallow: /checkout/
Disallow: /search
Allow: /search/help

Sitemap: https://www.example.com/sitemap.xml

• User-agent says which crawler the rules below it are for. * means all of them.
• Disallow lists the start of addresses not to fetch. Disallow: /checkout/ covers every address that starts with /checkout/.
• Allow makes an exception inside a disallowed part. Google follows the most specific rule: the longest matching one wins.
• Sitemap gives the full address of your sitemap. It can go anywhere in the file, and you can list more than one.
• Google understands * (any characters) and $ (end of the address) in rules. It ignores lines it does not understand, including Crawl-delay and Noindex.

The usual robots.txt mistakes

• Everything blocked. Disallow: / under User-agent: * blocks the whole site. It is often left over from a test version of the site. In Seoga’s scan it shows as “Search engines are blocked from your home page”, a critical finding.
• Styles and scripts blocked. Blocking folders that hold your CSS or JavaScript stops Google from seeing the page as visitors do: “Your robots.txt blocks the files your pages need”.
• Using robots.txt to hide pages. A blocked page can still appear in results as a bare address if other sites link to it. To keep a page out, let Google fetch it and add noindex.
• Typing mistakes. A missing colon or a misspelt word makes Google skip the line: “Your robots.txt file has mistakes”.
• No Sitemap line. Google can find your sitemap through Search Console, but other search engines rely on this line: “Your robots.txt doesn’t name your sitemap”.

1. Read the file line by line. For each Disallow, ask: do I really want search engines to stay out of every address that starts like this?
2. Test your important pages. In Search Console, the robots.txt report shows the file Google last fetched and any lines it could not read. URL Inspection tells you if a given page is blocked.
3. Edit and upload the file. Most website builders have a robots.txt setting or plugin; on your own hosting it is a plain text file in the site’s root folder. Keep it short.
4. Check the live file. Open it in the browser again. Google usually re-reads robots.txt within about a day; you can ask it to re-fetch the file from the robots.txt report.

When robots.txt is missing or fails

A site without a robots.txt file is not broken. If the address answers “not found” (404), Google treats it as having no rules and fetches everything it finds. Seoga still reports “Your site has no robots.txt file”, because the file is the natural place to name your sitemap and to keep crawlers out of pages like the checkout or internal search.

A server error is different. If robots.txt answers with a 5xx error, Google can’t tell what is allowed, and it slows down or pauses crawling the site until the file answers again. So if your Page indexing report suddenly shows fewer pages fetched, open the robots.txt address and check it loads. Hosting firewalls and security plugins sometimes block crawlers from this one file by mistake.

Test versions of a site cause the opposite problem. A copy kept for testing is often blocked with Disallow: /, which is right for the copy. When the copy becomes the live site, the block can travel with it. Check robots.txt on the day any new version of the site goes live.

What belongs in a sitemap

A sitemap should list only the pages you want people to find in search: pages that open with status 200, are not blocked, carry no noindex, and are the main copy of themselves. Every extra address in the sitemap is a small instruction Google has to check and then ignore.

<url>
  <loc>https://www.example.com/services/garden-design</loc>
  <lastmod>2026-09-30</lastmod>
</url>

• Use full addresses, written exactly the way the site uses them: the same https, the same www or not.
• Add <lastmod> only if it is the real date the page’s content last changed. Google uses it when it proves reliable, and ignores it when every page shows the same date.
• One sitemap file holds up to 50,000 addresses and 50 MB. Bigger sites split their pages into several files and list them in a sitemap index.
• Google ignores <priority> and <changefreq>, so there is no need to fine-tune them.

The usual sitemap mistakes

Most sitemap problems come from a sitemap that is built once and never kept in step with the site. Seoga’s scans compare every address in the sitemap with what the page really does, and report mismatches like these:

• “2 pages in your sitemap can’t be opened”: deleted or moved pages still listed.
• “1 page is in your sitemap but tells Google not to list it”: a sitemap entry and a noindex tag saying opposite things.
• “1 page is in your sitemap but blocked by robots.txt”: the same conflict with robots.txt.
• “3 pages in your sitemap point Google elsewhere”: addresses whose canonical tag names a different page. List the main copy instead.
• “Your sitemap uses a different form of your address”: http:// or the other www form, so every entry is a redirect.
• “Your sitemap hasn’t been updated recently”: a file generated once and forgotten.

1. Let the site build the sitemap. Most platforms and SEO plugins generate one that updates itself as pages change. A hand-made file goes stale.
2. Remove what doesn’t belong. Take out redirected, missing, blocked and noindex pages, and copies whose canonical names another page.
3. Name it in robots.txt. Add the Sitemap: line with the full address.
4. Submit it in Search Console. Open Indexing, then Sitemaps, enter the address and submit. The report shows when Google last read it and how many pages it found.