How to Make the Most of Your Crawl Budget

Peak is just around the corner for many of our clients, and in preparation, we have been working hard to make sure Google can crawl sites quickly and efficiently. We want Google to spend as much crawl budget as possible on ‘high-value’ pages, as opposed to low-value pages such as search parameter URLs (URLs created when users filter through options on site).

What have we found?

Looking at our top 10 retail e-commerce clients, we found that Google spends <10% of its crawl budget on the pages that we really want it to crawl (i.e. those that have been recently optimised). The biggest user of crawl budget tends to be Search parameter URLs which are always ‘low value’ from an SEO perspective, and often, we don’t want them to rank. Javascript, images and CSS also use up valuable crawl budget.

What did we do?

Firstly, we made changes to the directives we were giving to Google; we used the robots.txt file to block pages that were never going to deliver value, and made changes to the sitemap. Following on from this, we created static pages for popular search queries, thus reducing the amount of Search parameter URLs being created. Finally. we hosted images elsewhere (therefore reducing image crawl requests) and removed unnecessary Javascript and CSS.

What was the result?

On average, we increased the efficiency of crawl budget from <10% to over 30%, which is an excellent way to be heading into Peak!

Optimising Crawl Budget Usage: A Client Case Study

One of our largest e-commerce clients was experiencing issues with inefficient Google site crawls due to an unusually vast number of pages on their website. So, we got to work straight away to find a fix that would ensure that all crawl spend was being put to its best possible use! 

Here’s a brief snapshot of what we were up against and what we did about it;

The challenge

• The client has an extremely large site that has hundreds of thousands of product pages, which in itself poses a rare technical challenge. 
• Their previous agency (correctly) determined that Google had been spending all its time crawling the wrong pages, or it was taking too long re-crawling the most important pages.
• A decision was made by the previous agency to drop in a ‘no-index’ tag – this unfortunately worsened the issue as Google will still crawl pages that are marked as ‘no-index’, it just won’t index them.

The method

 We developed a rule-based system which gave clear directives to Google based on opportunity. 

• Rule 1: the page is ‘high value’ (we want it to rank highly) – for these pages we wrote unique content and included it in the sitemap.
• Rule 2: the page is ‘medium value’ (we want it to rank, but the page niche and SV is low) – here we implemented a canonical strategy.
• Rule 3: the page is very ‘low value’ – we blocked these pages via a robots.txt file and removed all ‘no-index’ tags.

The result!

• The site is now ranking for more pages than it ever has before!
• Post changes, 40% of high value pages were being crawled, compared to 5% pre changes
• There was a +24% increase in overall ranking keywords
• And a +32% increase of keywords on page 1 

The above results were seen within only a 20 day period, so it just goes to show, with the correct strategy in place, encouraging improvements are achievable in the blink of an eye – saving both time and valuable investment.

Get in touch if you’d like to hear more about this case study, or if you need some help with your crawl budget strategy.