Try Semrush

Semrush helps you:

  • Do keyword research
  • Audit your local listings
  • Perform competitor analyses
  • Manage social media accounts
  • And much more!

Backlinko readers get:

A 14-day trial for premium features. 55+ tools.
Free access for core features.

Newsletter Sign UpTry Semrush — Free Trial

Backlinko readers get unlimited
access for 14 days. 55+ tools.

Technical SEO

  • SEO Marketing Hub 2.0
  • Technical SEO
  • Crawl Budget
Sitemaps
Page Speed and SEO

Crawl Budget

What Is Crawl Budget?

Crawl Budget is the number of pages Googlebot crawls and indexes on a website within a given timeframe.

Why Is Crawl Budget Important for SEO?

In short, if Google doesn’t index a page, it won’t rank for anything, which could hurt your SEO performance.

So if your number of pages exceed your site’s crawl budget, you’re going to have pages on your site that aren’t indexed.

Very high page total might affect indexing

That said, the vast majority of sites out there don’t need to worry about crawl budget. Google and other search engines are REALLY good at finding and indexing pages.

That said, there are a few cases where you do want to pay attention to crawl budget:

  • You run a larger site: If you have a website (like an ecommerce site) with 10k+ pages, Google can have trouble finding them all.
  • You just added a bunch of pages: If you recently added a new section to your site with hundreds of pages, you want to make sure that you have the crawl budget to get them all indexed quickly.
  • Lots of redirects: Lots of unnecessary redirects and redirect chains eat up your crawl budget.

With that, here are some simple ways to maximize your site’s crawl budget.

Best Practices

Improve Site Speed

Improving your site’s page speed can lead to Googlebot and other search engine bots crawling more of your site’s URLs.

In fact, Google states that:

“Making a site faster improves the user experience while also increasing crawl rate.”

In other words:

Slow loading pages eat up valuable Googlebot time.

Slow loading pages are hard to crawl

But if your pages load quickly, Googlebot has time to visit and index more of your pages.

Quick load means more time to index other pages

Use Internal Links

Googlebot prioritizes pages that have lots of external and internal links pointing to them.

Yes, ideally you’d get backlinks pointing to every single page on your site. But that’s not a realistic internal link structure in most cases. Keep your pages optimized by eliminating any broken links. For internal links, broken links point to deleted or moved pages that haven’t been redirected/removed.

You need to find and fix broken links as a best SEO practice.

This shouldn’t overshadow the fact that internal linking is so key.

Your internal links send Googlebot to all of the different pages on your site that you want indexed.

Internal links send Googlebot to all pages you want indexed

Flat Website Architecture

According to Google:

“URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.”

And in the world of Google, popular=link authority.

That’s why you want to use a flat website architecture on your site.

A flat architecture sets things up so that all of your site’s pages have some link authority flowing to them.

Flat architecture lets link authority flow to all pages

Avoid “Orphan Pages”

Orphan pages are pages that have no internal or external links pointing to them.

Avoid unlinked orphan pages

Google has a really hard time finding orphan pages. So if you want to get the most out of your crawl budget, make sure that there’s at least one internal or external link pointing to every page on your site.

Limit Duplicate Content

Limiting duplicate content is smart for a lot of reasons.

As it turns out, duplicate content can hurt your crawl budget.

Factors affecting crawl budget

That’s because Google doesn’t want to waste resources by indexing multiple pages with the same content.

So make sure that 100% of your site’s pages are made up of unique, high quality content.

This isn’t easy for a site with 10k+ pages. But it’s a must if you want to get the most from your crawl budget.

Learn More

Optimize your crawling & indexing: A helpful guide to how Google finds, crawls and indexes pages.

Complete Guide to Crawl Budget Optimization: Super in-depth video on optimizing your crawl budget (includes real life examples).

Crawl Stats report (websites): A post from Google on how to read and interpret the Crawl report in the Google Search Console.

Next Page Speed and SEO
Previous Sitemaps
Next Page Speed and SEO
More Topics
All Topics
8 ResourcesSEO Fundamentals
4 ResourcesKeyword Research Strategies
8 ResourcesContent Optimization Strategies