Last updated
Why Google Not Indexing My Site
Google stops indexing a site when something between the crawler and the page actively tells it to stay out, not when content quality is the problem. That refusal sits in one directive layer, and finding which layer is issuing it determines the fix.
Site-wide blocks are different from page-by-page gaps
When Google stops indexing a site, the cause almost always sits in a directive or infrastructure layer between the crawler and the page, not in the writing itself. A noindex switch, a robots.txt rule, a canonical signal, or a hosting fault can each remove an entire domain from the index while every individual page still reads fine to a human visitor. This diagnostic isolates those four site-wide blockers so you stop rewriting content for a problem that content cannot fix.
This page focuses on blocks that hit the whole domain at once, not gaps that only affect part of it. If only some of your pages are missing from the index while most are fine, that is a narrower problem covered separately when only some of your pages are missing from the index. Everything downstream, including ranking and traffic, depends on this layer clearing first, which is also why crawl and index access sits as the first checkpoint on the SEO failure hub.
Six reasons Google stops indexing a whole site
Six patterns account for most site-wide indexing blocks. Each one can act alone or stack with another, so work through all six even after the first match explains part of what you are seeing.
A CMS 'discourage search engines' toggle noindexes every template
Most CMS platforms ship a single site-wide setting that inserts a noindex directive into the head of every page, independent of what any individual page’s SEO plugin says. WordPress calls it “Discourage search engines from indexing this site.” Other site builders bury an equivalent switch inside general or privacy settings. Teams turn it on during a staging build, a redesign, or a migration, and the launch checklist misses turning it back off.
Because the directive comes from one global setting instead of per-page metadata, individual page audits can look clean while the aggregate index count collapses across the whole domain. Check the CMS admin setting directly, then confirm in rendered HTML source that the meta tag is actually gone, since some platforms only stop injecting it after a full cache clear and republish.
What this looks like: Search Console lists most URLs as Excluded by noindex tag, and the source is a single CMS checkbox nobody remembers enabling.
A root-level Disallow: / rule blocks Googlebot from the entire domain
A single line in robots.txt can withdraw crawl permission for the whole site. Disallow: / under a wildcard user agent tells every well-behaved crawler, including Googlebot, to stay out of every path on the domain. This line often survives from a staging environment, a maintenance window, or a developer’s local testing setup, and it ships to production along with the rest of the deploy.
Unlike a path-specific disallow that only hides one section of a site, a root-level block removes the entire domain from future crawling, which eventually removes it from the index as previously crawled pages expire. Fetch robots.txt directly on the live domain and read it the way a crawler would, not through a plugin’s summary screen, since some plugins render a friendlier interpretation than the file that actually ships.
What this looks like: robots.txt returns Disallow: / under User-agent: *, and the Coverage report shows blocked by robots.txt across nearly every URL.
A sitewide canonical bug points every URL at one catch-all page
Canonical tags tell Google which URL should represent a set of duplicate or near-duplicate pages. A templating bug, a misconfigured plugin default, or a copy-pasted header snippet can make every page on a site declare the same canonical URL, regardless of what page actually loaded. Google respects that signal, consolidates the whole set toward the declared target, and drops the rest from the index.
This differs from a canonical loop between two URLs, which usually only confuses one pair of pages. A sitewide collapse removes the entire site’s individual URLs from consideration at once, because Google believes you told it there is only one page that matters. Compare the declared canonical against the requested URL on several unrelated templates to confirm the pattern before you change anything else.
What this looks like: View-source on ten unrelated templates all declares the identical canonical URL, usually the homepage or one leftover category page.
A hosting firewall blocks Googlebot's IP range while humans see nothing wrong
Web application firewalls, bot-management services, and some CDN security rules classify traffic by behavior and IP range, and an aggressive rule set can catch Googlebot in the same net set for scrapers. Because the block applies at the network edge, it can be completely invisible to a human tester browsing normally from a residential IP address; only automated fetchers from data-center ranges get refused.
The result looks like a content or quality problem from inside the CMS, because nothing about the page itself changed. Confirm the block by running Search Console’s live URL Inspection test and by fetching the URL with a Googlebot user agent from an external server, then check WAF and bot-management logs for the specific requests that were challenged or dropped.
What this looks like: URL Inspection's live test in Search Console times out or reports a fetch error, while the same URL loads normally in an ordinary browser tab.
A DNS cutover or expired SSL certificate breaks crawl access sitewide
Indexing depends on Googlebot being able to resolve the domain and complete a clean HTTPS handshake on every single fetch. A DNS migration that leaves old and new records live at the same time, a TTL that has not fully propagated, or a certificate that expired and was not renewed on the exact hostname being fetched can all interrupt that handshake intermittently rather than every time.
Intermittent failures are the hardest version of this problem, because the site looks fine on most checks and fails only on the requests that happen to hit the broken path. Test DNS resolution and the certificate chain from a server outside your own network and outside your usual browser cache, and check hosting-provider status pages for any migration or renewal event around the time indexing dropped.
What this looks like: A DNS lookup or certificate check from an external tool fails or returns inconsistent answers depending on which resolver or region makes the request.
An edge CDN worker injects X-Robots-Tag noindex before the CMS ever renders
Some indexing blocks never touch the HTML at all. A CDN edge worker, a reverse proxy rule, or a security header policy configured at the hosting layer can attach an X-Robots-Tag: noindex header to every response before the CMS finishes rendering the page. Search Console honors that header exactly like a meta tag, and it applies to every URL the rule matches, but editing the page’s on-page SEO settings will never remove it.
This block is easy to miss because most SEO plugins and page-level audits only read the HTML, not the raw response headers. Fetch a few live URLs with curl -I and read every header in the response, not just the status code, and check whatever edge or proxy layer sits in front of the origin server for a noindex rule left over from a staging configuration.
What this looks like: curl -I on the live URL returns an X-Robots-Tag: noindex response header, while the HTML body has no noindex meta tag anywhere in the source.
How a blocked crawl becomes a missing SERP slot
Indexing is not one event, it is a sequence, and a site-wide block usually stops the sequence at the very first step. Crawl has to succeed before anything else can happen, indexing has to accept the page before ranking systems ever see it, and only an indexed, ranked page becomes eligible for a specific SERP feature at query time.
Most of the confusion around “Google is not indexing my site” comes from judging the wrong stage. A page that never got crawled was never going to rank, no matter how strong its content is, and a page that does rank can still lose the visible slot to a stronger competitor at the final selection step. Indexing repair work sits with the rest of the crawl stack on content at scale, because index coverage and publishing velocity are measured on the same URLs.
From a blocked crawl to a lost SERP slot
- 01Googlebot attempts to fetch the URL, unless a directive or hosting fault refuses the request first
- 02The page enters the index only if the crawl succeeded and no noindex or canonical signal removed it
- 03Ranking systems score relevance and authority for the URLs that made it into the index
- 04Query-time systems select which SERP feature an already-ranked URL is eligible to fill
- 05The result surfaces for the query, or a competitor's clean URL wins the slot instead
| Blocker type | Where it lives | Search Console signal | Fastest verification |
|---|---|---|---|
| Sitewide noindex toggle | CMS setting or global meta tag | Excluded by noindex tag | View page source for the robots meta tag |
| Robots.txt wildcard block | robots.txt at the domain root | Blocked by robots.txt | Fetch robots.txt directly in a browser or curl |
| Canonical collapse | rel=canonical tag or HTTP link header | Alternate page with proper canonical tag | Compare the declared canonical to the URL you requested |
| Firewall or WAF block | Hosting edge, CDN, or bot-management layer | Crawled, currently not indexed | Run URL Inspection's live test and fetch with a Googlebot agent |
| DNS or SSL failure | DNS records or the certificate on the origin | Discovered, currently not indexed | Test resolution and the certificate chain from an external host |
| Edge header noindex | CDN worker or reverse proxy response headers | Excluded by noindex tag, invisible in HTML | Fetch the live URL with curl -I and read the headers |
Signs you have a site-wide block, not a content problem
Each sign below is testable directly, without guessing from ranking positions alone. Run through all eight before you conclude which layer is responsible, since more than one can be true at once.
SIGNS CHECKLIST
0 / 8 checked
How to clear a site-wide indexing block
Work through these in order. Confirming the block costs almost nothing, while hosting and DNS fixes take longer and should wait until you know they are actually the cause.
Fetch the site the way Googlebot fetches it
Fetch the homepage and key templates as Googlebot
Result: You know within one script run whether a noindex header, meta tag, or canonical mismatch is actively refusing entry.
- Pick the homepage plus one URL from every major template type
- Fetch each one with a Googlebot user agent and save the response headers separately from the HTML
- Check the saved headers for an X-Robots-Tag before you look at the HTML at all
- Check the HTML for a noindex meta tag and the declared canonical URL
#!/usr/bin/env bash
URL="https://www.example.com/"
curl -sSL -A "Googlebot/2.1" -D headers.txt -o page.html "$URL"
echo "--- X-Robots-Tag header ---"
grep -i "x-robots-tag" headers.txt || echo "none found in headers"
echo "--- Meta robots tag ---"
grep -io '<meta[^>]*name="robots"[^>]*>' page.html || echo "none found in HTML"
echo "--- Canonical tag ---"
grep -io '<link[^>]*rel="canonical"[^>]*>' page.html || echo "none found in HTML"
echo "--- robots.txt ---"
curl -sSL "https://www.example.com/robots.txt"
Cross-check Search Console's Page Indexing report against the fetch
Result: You confirm the fetch result matches what Google itself is reporting, instead of guessing from one tool alone.
- Open the Page Indexing report and sort by exclusion reason, not just total count
- Compare the top exclusion reason to what your curl fetch already found
- Run URL Inspection's live test on two or three affected URLs
- Note whether the last crawl date on affected URLs is recent or stale
Remove the directive that is refusing entry
Turn off the sitewide noindex toggle and confirm in rendered HTML
Result: The CMS-level directive stops emitting on every template, and the change is visible in a live fetch, not just the admin panel.
- Find and disable the global search-engine-visibility or noindex setting in the CMS
- Clear every layer of cache, including CDN and CMS-level caching, after the change
- Re-fetch the same URLs from step one and confirm the header and meta tag are both gone
- Check for a second, plugin-level noindex setting that can override the platform default
Rewrite robots.txt and align every canonical to itself
Result: robots.txt allows the crawl paths you need, and each URL's canonical points to itself unless a deliberate consolidation is intended.
- Replace any root-level Disallow: / with only the specific paths that should stay private
- Add a Sitemap line pointing at your current sitemap
- Fix the canonical template so each page defaults to referencing its own URL
- Reserve cross-URL canonicals for genuine duplicates, and document why each one exists
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
# Remove any line that reads: Disallow: /
# Remove duplicate or conflicting User-agent and Sitemap blocks
Fix the hosting fault, then ask Google to recrawl
Clear the firewall, WAF, or DNS fault blocking Googlebot
Result: Googlebot's fetch requests complete successfully from outside your own network on every attempt, not just most of them.
- Ask your hosting or security team to allowlist verified Googlebot IP ranges and user agents
- Review WAF and bot-management logs for challenged or blocked requests during the affected window
- Confirm DNS records point to the correct current origin with no stale or duplicate entries
- Renew or reissue the SSL certificate for every hostname the site actually serves
Request indexing and monitor recovery for six weeks
Result: You have a clear before-and-after record instead of judging recovery from memory or a single screenshot.
- Submit the homepage and a handful of key templates through URL Inspection's request indexing tool
- Resubmit the sitemap after all directive and hosting fixes are live
- Track the Page Indexing report weekly rather than daily, since recrawl takes time
- Re-run the AEO score checker once indexing recovers to confirm AI-facing visibility follows
What a sitewide block does to your AI visibility score
VISIBILITY INSIGHT
A blocked domain scores near zero on every AI visibility dimension at once
An AI visibility score blends mention frequency, citation share, factual accuracy, entity strength, and competitive share of voice across engines. A site with a sitewide indexing block scores low on every one of those dimensions simultaneously, because a domain absent from Google's index rarely builds the crawl history and citation trail models draw on either. SearchDock tracks whether your indexed pages are actually being fetched and cited across ChatGPT, Perplexity, Google AI Overviews, Gemini, Claude, and Copilot, separate from classic Search Console coverage. It also flags when a domain's score stays flat despite a reported indexing fix, which usually means the block only partially cleared.
See where your AI visibility score breaks downA fixed indexing block is necessary but not sufficient, since AI engines still need their own crawl and citation signals confirmed separately.
Related indexing and SEO diagnostics
These pages cover the narrower and adjacent symptoms that a sitewide block can resemble.
Fix the block, then everything else can work
Fix the block before you fix anything else, because no amount of content work, link building, or schema polish can matter to a domain that never enters the index. Start with a live fetch, not the CMS admin screen, since headers and canonical tags tell the truth that settings pages can hide. Clear the directive layer first, then the hosting layer, and re-test with a fresh fetch after every change. If the whole domain still looks invisible in search after these fixes, also check why your website is not on Google at all for the broader diagnostic.
See whether Google and AI engines can both reach your siteFrequently asked questions
Why is Google not indexing my site at all?
Google is not indexing your site when a directive or infrastructure fault refuses the crawler entry before content quality ever gets evaluated. The usual causes are a sitewide noindex toggle, a robots.txt rule blocking the whole domain, a canonical bug collapsing every URL into one, or a hosting fault such as a firewall, DNS, or SSL failure.
How do I know if noindex is blocking my whole site?
Fetch the homepage with a Googlebot user agent and check both the response headers and the HTML for a noindex directive. If Search Console's Page Indexing report shows most URLs excluded for the same reason, and a CMS setting like discourage search engines is enabled, a sitewide noindex toggle is the likely cause.
Can a canonical tag stop my whole site from being indexed?
Yes, when a templating bug makes every page declare the same canonical URL instead of referencing itself. Google treats that as an instruction to consolidate the entire set toward one destination, which removes the other URLs from the index even though each page still loads correctly for a normal visitor browsing the site.
Why does robots.txt block my entire site instead of just certain paths?
A wildcard Disallow: / line under User-agent: * withdraws crawl permission from every path at once, unlike a path-specific rule that only hides one section. This line often survives from a staging environment or maintenance window and ships to production by accident, so check the live file directly rather than a plugin summary.
Can hosting or a firewall stop Google from indexing my site?
Yes. A web application firewall, bot-management rule, or CDN security setting can block Googlebot's requests by IP range or behavior pattern while a normal browser still loads the page fine. Confirm this with Search Console's live URL Inspection test and a Googlebot-agent fetch run from a server outside your own network.
How long does it take Google to re-index my site after I fix a block?
Recovery usually starts within days of the next crawl, but full re-indexing across a whole site can take several weeks depending on crawl budget and domain size. Request indexing on key URLs after the fix, resubmit your sitemap, and track the Page Indexing report weekly instead of judging recovery from one screenshot.
Does fixing indexing also help my visibility in AI answers?
It helps but does not guarantee it. AI engines run their own crawlers and citation logic on top of whatever Google indexes, so a domain that regains classic indexing still needs separate crawl access and content signals confirmed for those systems before it starts appearing inside AI-generated answers.