How to check crawling and indexing

How a page gets into Google

Before a page can show up in search results, Google has to do four things with it. It has to find the address. It has to fetch the page, which is called crawling. It has to read and understand what is on it, running any scripts the page needs. And it has to decide to keep the page in its index, the huge list of pages it can show. Only then does ranking start.

Each step can fail on its own, and each failure looks the same from the outside: the page is simply not there. Checking crawling and indexing means finding out which step failed.

A quick first look

Search Google for site:yourdomain.com. You will see a sample of the pages Google has for your site. It is a rough sign only: Google says the count is an estimate and the list is not complete. If your home page or main pages are missing, though, you have found a problem worth chasing.

For a real answer, use Search Console. Two parts of it do most of the work: the Page indexing report for the whole site, and the URL Inspection tool for one page at a time.

Read the Page indexing report

In Search Console, open Indexing, then Pages. The chart splits your known pages into indexed and not indexed. Below it, the “Why pages aren’t indexed” table groups the left-out pages by reason. Not every reason is a fault. A page you redirected on purpose belongs there. The work is to find reasons that cover pages you want in Google.

Check one page by hand

When an important page is missing, check it directly. You will need the page address and a browser.

1. Inspect the address in Search Console. Paste it in the search bar at the top. The result says whether the URL is on Google and, if not, gives the same reasons as the report above.
2. Look at what Google saw. Choose “Test live URL”, then “View tested page”. The screenshot and the HTML show the page as Google reads it. If your main text is missing there, Google can’t read it, often because it only appears after a script runs.
3. Check the status code. In your browser, open the developer tools, then the Network tab, and reload the page. The first line should show status 200. A 301 or 302 means the page redirects; a 404 or 5xx means it fails.
4. Look for a noindex. View the page source and search for noindex. It can sit in a <meta name="robots"> tag, or in an X-Robots-Tag header that you can see in the Network tab.
5. Check the canonical tag. Search the source for rel="canonical". It should hold the page’s own address, written the same way you link to it (same https, same www or not, same ending).
6. Ask Google to look again. Once the page is fixed, use “Request indexing” in URL Inspection. This puts it in Google’s queue; it does not promise when, or whether, it will be listed.

Common causes, as a scan reports them

Seoga’s scans check these causes on every page they read. In a scan report they read like this:

• “2 pages tell Google not to list them”: a noindex left over from building the site, or a setting in the website builder that was never switched back.
• “1 page is in your sitemap but blocked by robots.txt”: two files giving Google opposite instructions. The sitemap says “read this”, robots.txt says “don’t”.
• “4 pages have no links pointing to them”: pages only reachable through the sitemap. Google may find them, but sees little reason to rate them.
• “3 pages are too many clicks from the home page”: pages so deep in the site that they are visited rarely.
• “1 page points Google to a different page”: a canonical tag that names another address, often copied from a template by mistake.

If several important pages are missing, look for one shared cause first: a template, a plugin setting, or one robots.txt rule. One fix there can bring back many pages at once.

Do you need to worry about crawl budget?

Google spends a limited amount of time on each site. For most sites that limit never matters. Google’s own guide says crawl budget is a concern mainly for very large sites, with about a million pages or more, or for sites with tens of thousands of pages that change every day. If your site is smaller, a page that is not crawled is almost always a page with too few links to it, or one Google sees little value in.

Large sites can still waste Google’s visits. Filter pages that create endless combinations, addresses with session codes, and long chains of redirects all make Google fetch pages that add nothing. Seoga’s scans flag these as, for example, “Your filters create many copies of the same pages”.