Crawl Budget
Written by Brian Dean
What Is Crawl Budget?
Crawl Budget is the number of pages Googlebot crawls and indexes on a website within a given timeframe.
Why Is Crawl Budget Important for SEO?
In short, if Google doesn’t index a page, it won’t rank for anything, which could hurt your SEO performance.
So if your number of pages exceed your site’s crawl budget, you’re going to have pages on your site that aren’t indexed.
That said, the vast majority of sites out there don’t need to worry about crawl budget. Google and other search engines are REALLY good at finding and indexing pages.
That said, there are a few cases where you do want to pay attention to crawl budget:
- You run a larger site: If you have a website (like an ecommerce site) with 10k+ pages, Google can have trouble finding them all.
- You just added a bunch of pages: If you recently added a new section to your site with hundreds of pages, you want to make sure that you have the crawl budget to get them all indexed quickly.
- Lots of redirects: Lots of unnecessary redirects and redirect chains eat up your crawl budget.
With that, here are some simple ways to maximize your site’s crawl budget.
Best Practices
Improve Site Speed
Improving your site’s page speed can lead to Googlebot and other search engine bots crawling more of your site’s URLs.
In fact, Google states that:
“Making a site faster improves the user experience while also increasing crawl rate.”
In other words:
Slow loading pages eat up valuable Googlebot time.
But if your pages load quickly, Googlebot has time to visit and index more of your pages.
Use Internal Links
Googlebot prioritizes pages that have lots of external and internal links pointing to them.
Yes, ideally you’d get backlinks pointing to every single page on your site. But that’s not a realistic internal link structure in most cases. Keep your pages optimized by eliminating any broken links. For internal links, broken links point to deleted or moved pages that haven’t been redirected/removed.
You need to find and fix broken links as a best SEO practice.
This shouldn’t overshadow the fact that internal linking is so key.
Your internal links send Googlebot to all of the different pages on your site that you want indexed.
Flat Website Architecture
“URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.”
And in the world of Google, popular=link authority.
That’s why you want to use a flat website architecture on your site.
A flat architecture sets things up so that all of your site’s pages have some link authority flowing to them.
Avoid “Orphan Pages”
Orphan pages are pages that have no internal or external links pointing to them.
Google has a really hard time finding orphan pages. So if you want to get the most out of your crawl budget, make sure that there’s at least one internal or external link pointing to every page on your site.
Limit Duplicate Content
Limiting duplicate content is smart for a lot of reasons.
As it turns out, duplicate content can hurt your crawl budget.
That’s because Google doesn’t want to waste resources by indexing multiple pages with the same content.
So make sure that 100% of your site’s pages are made up of unique, high quality content.
This isn’t easy for a site with 10k+ pages. But it’s a must if you want to get the most from your crawl budget.
Learn More
Optimize your crawling & indexing: A helpful guide to how Google finds, crawls and indexes pages.
Complete Guide to Crawl Budget Optimization: Super in-depth video on optimizing your crawl budget (includes real life examples).
Crawl Stats report (websites): A post from Google on how to read and interpret the Crawl report in the Google Search Console.