How to Fix Common Crawl Errors and Soft 404s in Google Search Console

The first time I opened the Page indexing report on one of my WordPress sites, I saw a grey bar labelled “Not indexed” that was bigger than the green one. Hundreds of URLs. Soft 404s, redirect errors, pages “crawled but not indexed.” It looked like the site was broken.

It wasn’t. Most of those URLs were never meant to be indexed in the first place. A handful, though, were genuine problems quietly costing me traffic. The skill isn’t fixing every line in that report — it’s knowing which errors matter and which ones are just Google keeping you informed.

This guide walks through every common crawl error you’ll see in Google Search Console, what each one actually means, and the exact steps I use to fix them on WordPress. If you haven’t connected Search Console to your site yet, start with our Google Search Console integration guide for WordPress and come back.

First, Understand What “Crawl Errors” Mean in 2026

Google retired the old “Crawl Errors” report years ago. Today the same information lives in two places:

• Pages (Page indexing report): Shows which URLs are indexed and, more importantly, why other URLs aren’t.
• Settings → Crawl stats: Shows how often Googlebot visits, how fast your server responds, and what share of requests return errors.

Google’s own Page indexing report documentation lists every possible status. The key mindset shift: “Not indexed” is not the same as “broken.” A tag archive excluded by noindex is working exactly as designed.

The Three Questions I Ask for Every Error

1. Should this URL be indexed at all? If not, the error may be harmless.
2. Does this URL get traffic or links? If yes, fix it first.
3. Is it a pattern? One broken URL is a typo. Five hundred broken URLs is a template, plugin or setting problem.

Quick Reference: Common Errors and What to Do

How to Fix Soft 404 Errors

Soft 404s are the most misunderstood error in Search Console, so let’s start there.

What a Soft 404 Actually Is

A normal 404 tells Google, “This page doesn’t exist.” A soft 404 happens when your server says 200 OK (the page exists), but the content looks like an error or an empty page to Google. Google treats it as missing anyway and drops it from the index — but it still wastes crawl effort on it.

The Most Common Soft 404 Causes on WordPress

• Empty category, tag or search pages — “No posts found” with a 200 status.
• Out-of-stock WooCommerce products with almost nothing left on the page.
• Thin pages — a heading and one sentence, often auto-generated.
• Custom “not found” templates that a theme serves without sending a real 404 status.
• Redirecting deleted posts to the homepage. Google often flags irrelevant redirects like this as soft 404s.
• Pages that rely on JavaScript to load content that fails to render for Googlebot.

Step-by-Step Soft 404 Fix

1. Export the list. In Pages, click “Soft 404” and export the affected URLs.
2. Group them by pattern. Are they all tags? All products? All under one folder? The pattern tells you the fix.
3. Test a sample with URL Inspection. Click “Test live URL,” then “View tested page” to see the screenshot and HTML Google actually received.
4. Choose the right action for each group:
   • Page should exist and has value → add genuine, useful content.
   • Page is permanently gone → return a real 404 or 410.
   • Page moved, and a close equivalent exists → 301 redirect to that specific equivalent (not the homepage).
   • Page is useful for users but not for search (empty tags, internal search) → add noindex.
5. Validate the fix. Back in Search Console, click “Validate fix” so Google re-checks the group.

My Rule for Out-of-Stock Products

Temporarily out of stock? Keep the page live with full product details, an “expected back” note, and links to similar products. Permanently discontinued? Redirect to the closest replacement or the parent category. Never leave a near-empty page returning 200 OK. Our WooCommerce SEO guide covers product-page handling in more depth.

How to Fix “Not Found (404)” Errors

Here’s something many site owners don’t believe until they hear it from Google: 404 errors are normal and don’t hurt your rankings on their own. Deleted pages, old URLs and typos in other people’s links all produce them.

Only spend time on 404s that meet at least one of these conditions:

• The URL has backlinks from other websites (check in Ahrefs Webmaster Tools or Search Console’s Links report).
• The URL used to get organic traffic.
• Your own site still links internally to it.

Fixing Valuable 404s

1. Find the closest relevant live page.
2. Add a 301 redirect from the dead URL to that page.
3. Update any internal links so they point directly to the new URL instead of going through the redirect.

On WordPress, you don’t need .htaccess edits for this — a redirection manager handles it cleanly. I explain the setup in our redirection manager guide. To catch internal links pointing to dead pages before Google does, run a broken link checker once a month.

How to Fix Server Errors (5xx)

Server errors are the ones I treat as urgent. A 5xx means Googlebot showed up and your server couldn’t deliver the page. If it happens often, Google automatically slows its crawl rate to avoid overloading your site — which means new content gets discovered more slowly.

Common Causes

• Shared hosting hitting resource limits during traffic spikes or heavy crawling
• A plugin causing PHP fatal errors or timeouts
• Security plugins or firewalls blocking or rate-limiting Googlebot
• Database connection limits being reached
• Maintenance mode left switched on

How I Diagnose Them

1. Open Settings → Crawl stats and look at “By response.” Check the dates when 5xx errors spiked.
2. Compare those dates with your hosting’s error logs and any plugin updates you made.
3. Inspect a failed URL live. If it loads now, the problem was intermittent — usually resources or a firewall.
4. Make sure your firewall or CDN isn’t challenging verified Googlebot traffic.

Google’s guide to how HTTP status codes and network errors affect crawling is worth reading once — it explains why repeated 5xx responses lead Google to back off.

If errors keep returning even after you’ve cleaned up plugins, it may simply be time for better hosting. We tested providers in our best WordPress hosting 2026 comparison.

How to Fix Redirect Errors

“Redirect error” means Googlebot followed a redirect and couldn’t reach a final page. The usual suspects:

• Redirect loops: Page A → Page B → Page A.
• Long chains: A → B → C → D → E. Keep it to one hop wherever possible.
• Redirects to a broken or empty URL.
• Conflicting rules between your SEO plugin, a redirection plugin, your server config and your CDN — especially HTTP→HTTPS and www→non-www rules stacked on top of each other.

The Fix

1. Crawl the affected URLs with Screaming Frog and open the “Redirect Chains” report.
2. Point every old URL straight at the final destination in one 301.
3. Keep redirect rules in one place. If your host already forces HTTPS, don’t add the same rule in a plugin too.

“Blocked by robots.txt” and “Excluded by noindex”

These two are usually intentional, but accidents happen more than you’d think — a staging setting copied to live, or the WordPress “Discourage search engines” box left ticked.

Check robots.txt

• Open yoursite.com/robots.txt and make sure you’re not blocking important folders, CSS or JavaScript.
• Remember: robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it.
• To keep a page out of the index, allow crawling and use noindex instead.

For the correct WordPress setup, follow our robots.txt guide.

Check Noindex

• Go to Settings → Reading in WordPress and confirm “Discourage search engines from indexing this site” is unchecked.
• Review your SEO plugin’s global robots meta settings for posts, pages and categories.
• Spot-check important pages with URL Inspection or a browser extension that shows robots tags.

Duplicate Content and Canonical Issues

“Duplicate without user-selected canonical” and “Duplicate, Google chose different canonical than user” mean Google found several versions of the same content and had to pick one itself.

On WordPress, the common sources are:

• URL parameters (tracking tags, sorting, filters)
• Attachment pages for images
• HTTP and HTTPS, or www and non-www, both resolving
• Paginated archives and print versions

The fix is to set clear self-referencing canonicals, redirect duplicate host variants, and disable attachment pages (most SEO plugins can redirect them to the parent post). Our guide to SEO canonical URLs in WordPress walks through every setting.

“Crawled – Currently Not Indexed” and “Discovered – Currently Not Indexed”

These aren’t technical errors in the strict sense, but they’re the statuses site owners ask me about most.

Crawled – Currently Not Indexed

Google fetched the page and decided it wasn’t worth indexing — for now. This is usually a quality or uniqueness signal, not a technical one. What helps:

• Expand thin content with real experience, examples and data.
• Merge several weak, overlapping posts into one strong one, and redirect the rest.
• Add internal links from relevant, already-indexed pages.
• Make sure the page answers a query people actually search for.

Discovered – Currently Not Indexed

Google knows the URL exists but hasn’t crawled it yet. On a small site this usually clears up on its own. If it persists, the causes are usually weak internal linking, a slow server, or too many low-value URLs competing for crawl attention. Clean sitemaps, faster responses and stronger internal links all help.

Using the URL Inspection Tool the Right Way

URL Inspection is the most useful debugging tool in Search Console, yet most people only use its “Request indexing” button. Here’s how I actually use it when an error doesn’t make sense:

1. Check the indexed version first. The default view shows what Google stored last time. Note the “Last crawl” date and the “User-declared” vs “Google-selected” canonical.
2. Run “Test live URL.” This fetches the page right now, so you can tell whether a problem has already been fixed.
3. Open “View tested page.” Look at the screenshot and the rendered HTML. If the main content is missing, Googlebot isn’t seeing what your visitors see — a classic cause of soft 404s on JavaScript-heavy pages.
4. Check “More info.” The resources list shows blocked or failed CSS and JS files that can break rendering.
5. Request indexing only after fixing. Requests are limited each day, so save them for important pages that you’ve genuinely improved.

Google’s help page on the URL Inspection tool explains every field in the report.

WordPress-Specific Crawl Error Traps

Some crawl errors are almost unique to WordPress. These are the ones I find most often when auditing other people’s sites:

• Attachment pages: Every uploaded image can get its own thin URL. These often show up as soft 404s or duplicates. Redirect attachment pages to the parent post or the file itself.
• Author archives on single-author blogs: They duplicate the main blog feed. Noindex them or disable them.
• Old permalink structures: If you changed from /?p=123 or date-based URLs, make sure old URLs redirect in one hop to the new ones.
• Paginated comment pages: /comment-page-2/ URLs create near-duplicates. Disable comment pagination unless you really need it.
• Feed URLs: /feed/ URLs sometimes appear as “Crawled – currently not indexed.” That’s normal and safe to ignore.
• Plugin-generated pages: Form confirmation pages, print views and AMP leftovers from removed plugins frequently become 404s or soft 404s.

A Real Cleanup Example

On one content site, the Page indexing report showed a few hundred soft 404s. Grouping the URLs revealed that almost all of them were tag archives containing a single post each — WordPress had created a new tag every time a writer typed a slightly different keyword.

The fix took about an hour:

1. Merged near-duplicate tags into a small, consistent set.
2. Set the remaining tag archives to noindex and removed them from the sitemap.
3. Redirected deleted tag URLs to the most relevant category.
4. Clicked “Validate fix” on the soft 404 group.

Within a few weeks, the soft 404 group had largely cleared, and Googlebot’s crawl activity in Crawl stats shifted towards actual articles. No new content was written — the site simply stopped asking Google to look at pages that never deserved attention.

A Monthly Crawl Health Routine (15 Minutes)

Here’s the routine I follow on the first Monday of every month for each site:

1. Pages report: Note the indexed count. Any sudden drop deserves a look.
2. Check new error groups: Soft 404, 5xx and redirect errors first.
3. Crawl stats: Average response time trending up? Host or plugin issue brewing.
4. Sitemaps report: Confirm every sitemap shows “Success” and a recent read date.
5. Run a crawl with Screaming Frog or your SEO plugin’s audit to catch broken internal links.
6. Validate fixes for anything you resolved last month.

For a broader checklist that goes beyond crawling, use our full SEO site audit guide.

Mistakes to Avoid When Fixing Crawl Errors

• Redirecting every 404 to the homepage. Google often treats this as a soft 404 anyway, and visitors hate it.
• Panicking over excluded URLs. Excluded doesn’t mean broken. Many exclusions are healthy.
• Blocking pages in robots.txt to “remove” them. Use noindex or a 404/410 instead.
• Clicking “Validate fix” before actually fixing. Failed validations just delay the re-check.
• Requesting indexing for hundreds of URLs one by one. Fix the cause, update your sitemap, and let Google recrawl naturally.

Frequently Asked Questions

Do soft 404 errors hurt SEO?

They don’t trigger a penalty, but the affected pages won’t be indexed, and a large number of them wastes crawl activity that could go to your important pages.

How long does “Validate fix” take?

It typically takes from a few days to around two weeks, depending on how many URLs are in the group and how often Google crawls your site.

Should I return 404 or 410 for deleted pages?

Both work. A 410 (“Gone”) may get a URL dropped slightly faster, but the practical difference is small. Use whichever your setup makes easier.

Why does a page work in my browser but show a 5xx in Search Console?

Usually because the error was temporary — a resource spike or a firewall rule that treats bots differently. Use “Test live URL” to see what Googlebot gets right now.

Is “Page with redirect” an error?

No. It simply means the URL redirects somewhere else, so Google indexes the destination instead. Just make sure your sitemap lists final URLs, not redirecting ones.

Final Thoughts

Search Console’s indexing report looks scary until you learn to read it as a to-do list rather than a list of failures. Fix server errors immediately, clean up soft 404s by pattern, redirect only the 404s that carry real value, and keep your robots, noindex and canonical settings deliberate.

Do that consistently, and the grey “Not indexed” bar stops being a worry and starts being a sign that your site is telling Google exactly what you want it to. For official reference whenever you’re unsure, Google’s crawling and indexing documentation is the final word.