Getting your pages crawled properly is the first step to appearing in Google’s search results. If Googlebot can’t find or access your website, you won’t show up when potential customers search for your products or services.
You could have the best content in the world, but if Googlebot can’t crawl it properly, it’s like having an amazing shop in the middle of nowhere with no roads leading to it.
That’s why understanding the basics of how Googlebot works is so important – it helps you make better decisions about your website and by the end of this video, you’ll fully comprehend what crawling is and how Googlebot works.
What Is Googlebot?
Googlebot is a web crawler, which is an automated program that explores the internet. Its job is to visit websites, find pages, and collect their content so Google can process it and decide how to show it in their search results.
This process of discovering pages is called “crawling”.
Googlebot primarily finds pages by following links from one page to another, just like a person clicking through a website. It gathers information from the page such as the text, images, and code and sends this data back to Google’s systems for further analysis.
Most websites are now crawled using the mobile version of Googlebot, called Googlebot Smartphone. This means Google looks at how your site works on a phone rather than a computer.
And that’s why it’s so important to make sure your site works well on mobile devices.
How Does Googlebot Find And Crawl Pages?
There are a few ways Googlebot finds pages so let’s go through the most common ones:
The First Is Following Links
As I mentioned earlier, the main way Googlebot discovers new pages is by following links. Just like how you might click links to move from one page to another, Googlebot follows links to find new content.
These could be internal links, which connect one page of your website to another, or external backlinks, which come from other websites linking to your site.
An important takeaway here is that if you have pages on your website that aren’t linked from anywhere else, Googlebot might never find them. That’s why we’ll take a close look at internal links later in this course.
Another Way Googlebot Finds Pages Is Through Sitemaps
An XML sitemap is exactly what it sounds like, a map of your website. It’s basically a list of all the pages you want Google to know about.
You can submit your sitemap through Google Search Console, and if your website has an automated sitemap, which is common with most CMS platforms, it will automatically update with new pages.
This is especially helpful for new pages that don’t yet have any links pointing to them yet, which are known as “orphan pages”, as it allows Google to find them quickly.
You Can Also Force The Page To Be Crawled Through Google Search Console
All you have to do is go into the URL Inspection tool, paste the page’s URL, hit enter and then click request indexing.
This is extremely useful if you’ve made a key update to a page, or you’ve launched a new page to make sure it gets crawled quickly.
How Internal Links Show Google That A Page Is Important
Internal links are more than just a way for Googlebot to find pages – they’re also a signal of importance. Think of them like votes of confidence from your own website.
When you link to a page multiple times across your website, you’re essentially telling Google “this page is important, and that’s why we keep referring to it”.
Imagine you have a website selling various types of coffee machines.
If your blog posts, product categories, and other pages frequently link to your “Best Coffee Machines for Offices” page, Google understands that this is likely a significant page for your website.
However these links need to make sense for your users. Don’t just add internal links for the sake of it. They should enhance the user experience by providing relevant, helpful connections between related content.
What You Need To Know About Crawl Budget
Crawl Budget is the number of pages Googlebot will crawl on your website within a given timeframe. Google doesn’t have unlimited resources, so it needs to be selective about how many pages it crawls on each website.
If you’re running a small to medium-sized website (let’s say under 10,000 pages), you probably don’t need to worry too much about crawl budget. Google is generally quite good at crawling these sites efficiently.
However, if you’re managing a larger website, crawl budget becomes more important.
For example:
- If your website has over 10,000 pages
- If you’re regularly adding lots of new content
- If you have loads of dynamic pages (like on an e-commerce site with lots of filtering options)
- If you have a significant amount of URLs marked in Search Console as “Discovered – currently not indexed”.
- Or if your server is slow or struggles with too many requests
In those instances, you might want to look a bit further into this topic.
However, for most websites, crawl budget isn’t something you need to be overly concerned about.