Crawlability

What does it mean for your website to be crawlable? The term “crawlability” relates to how easy search engine bots can access, read, and move through the pages on your site.

Before any page can rank, it needs to be found and read by a crawler like Googlebot, which works by following links from page to page and analysing the content it discovers. If something gets in the way of this process, a page may stay uncrawled forever, and if it can’t be crawled, it won’t be indexed or ranked.

Because of this, crawlability is one of the most important factors that drives high-quality technical SEO.

Contrasts with Indexability

It can be useful to understand the differences between crawlability and indexability. While crawlability is about whether a search engine can reach and read webpages, indexability is more about whether, having already crawled it, a search engine is allowed and able to store it in its index.

Pages that are crawlable aren’t necessarily guaranteed to be indexable and can be deliberately left out of the index with a noindex tag. They can also be blocked from crawling entirely so it can potentially never even get as far as indexing. Both have to go right for a page to appear in search results.

Damaging Crawlability

There are a few factors that can quietly harm your website’s crawlability. One example is a robots.txt file that accidentally blocks important sections or content that’s buried so deep in a site that crawlers can hardly reach it. Orphan pages with no internal links pointing to them, broken links that lead crawlers to dead ends, and a heavy reliance on JavaScript that bots struggle to render can all damage the crawlability of pages.

Poor site structures are a common culprit when it comes to investigating why your pages are struggling with their crawlability. If your important pages are many clicks from your homepage without enough internal links, there’s a risk that your crawlers won’t visit them enough to help with their indexing.

Importance of Crawlability

Good crawlability is all about getting the fundamentals right on your website. With the help of a logical, shallow site structure, where important pages are easy to reach, stronger internal linking assist crawlers to find the pages that you would prefer to be indexed, ensuring that you rank for the most valuable areas of your website.

Clean XML sitemaps that list your key URLs and a sensible robots.txt that blocks only what genuinely should be blocked can also help.

When exploring your crawlability, Google Search Console should be your first port of call, with its crawl stats and coverage reports showing what the search engine can and can’t reach.

Other crawling tools like Screaming Frog can simulate how a bot moves through your site and surface pages that are hard to reach or may be blocked.