Crawl budget sounds like a Google problem. For most sites it is a site problem: too many URLs that look like pages, too many redirects, and a sitemap full of things that should never have been indexed. The fix is boring and it is usually on your side of the fence.
The spider crawling tool fetches a URL twice, once as Googlebot and once as a browser, and compares status, headers, canonical, robots rules and the text that comes back. It is not Google's actual crawl. It is a way to see what a bot sees, which is the part you can control.
What a bot sees that you do not
- JavaScript shell. The browser renders the page; the bot may get an empty shell if the content is client-rendered and the crawl is delayed.
- Blocked resources. A robots.txt rule that blocks a script or a stylesheet can hide the page's content from rendering.
- Wrong canonical. The canonical that looks fine in the browser can be injected by JavaScript and missed by the crawler.
- Firewall blocks. A WAF rule may return 403 to a bot user agent while serving the page to you.
- Soft 404s. A page that returns 200 with "not found" text is worse than a real 404 because the bot keeps coming back.
The crawl-budget leaks
Every URL that returns a 200, a redirect or a soft 404 uses crawl attention that could go to the pages you want indexed. The common leaks:
- Faceted navigation. Filters and sort parameters create thousands of near-duplicate URLs. Most should be blocked or consolidated with a canonical.
- Search pages. Internal search results should almost always be blocked. They are infinite and low quality.
- Tag and archive pages. Thin archives with two posts each add URLs and little value.
- Redirect chains. Three hops for one link means three requests for one destination.
- Broken links and 404s. A site full of dead links teaches the bot that the site is unreliable.
What to check this week
Open the spider crawl and fetch the homepage, the main service page and one article. Compare what the bot gets with what the browser gets. If the title, canonical or main text differ, you have a rendering problem, and the fix is to make the important content server-rendered or prerendered.
Then open the sitemap and count the URLs. If the sitemap has 5,000 URLs and the site has 200 real pages, the sitemap is lying. Trim it to the pages you want indexed and let the rest be discovered by links.
Finally, run the broken links finder on the top pages. Dead internal links are crawl waste and a bad experience at the same time, which is the rare problem where the SEO fix and the user fix are identical.
When crawl budget is not your problem
If the site has fewer than 500 pages and a clean sitemap, crawl budget is probably not the explanation for a ranking problem. The page may simply not answer the query, or the query may not have volume. Do not spend a month on crawl configuration when three pages need better titles.
The honest way to decide: compare the pages that get indexed to the pages that get impressions. If every page is indexed and most have no impressions, the problem is relevance, not crawling. If half the site is not indexed, start with the crawl.
What to fix first
Robots.txt and the sitemap first, because they are text files and mistakes are visible. Then the canonical tags on the pages that matter. Then the redirect chains. Then the faceted navigation, which is a bigger project and usually needs a developer.
The crawl checklist
Work through this list in order. It catches the problems that matter before you spend a day on configuration.
- Robots.txt. Fetch it. Does it block anything it should not, and does it declare the sitemap?
- Sitemap. Does every URL return 200 and a canonical that points at itself? Remove anything that redirects or 404s.
- Indexability. Sample five pages and check for noindex, canonical mismatches and soft 404s.
- Bot view. Fetch the top page as a bot and compare the title, canonical and main text with the browser version.
- Redirects. Find chains of two or more hops. Point internal links straight at the final URL.
- Parameter URLs. Search the crawl for "?" and decide whether those URLs should be indexed, canonicalised or blocked.
- Log check. If the host gives you logs, look at which URLs Googlebot actually requests and how often it gets a 404.
That last item is the one most small sites never do, and it is the most informative. The log shows the crawl you actually got rather than the crawl you assumed. Five minutes of reading it beats a week of guessing.
One number worth checking: the ratio of indexed pages to pages that get impressions. If most of the index has never appeared in a search result, the site is carrying pages that exist for no one, and the sitemap is the easiest place to start pruning.
Bookmark the crawl. You will run it again next month, and the second run is where the improvement shows.
If you want the full picture, the website auditor checks the single page, the malware scan rules out an infection that could be blocking you. The tools hub gathers all of them. If the crawl problem needs a developer and you do not have one, the service page covers the fix.