Transcript
We briefly touched on site maps in an earlier video when we talked about how Googlebot discovers pages, but in this lesson, we’re going to take a much deeper look at them.
We’re going to find out exactly what they are, the different types of sitemaps, how they help with indexation, and how to submit them to Google.
Now, while they’re especially important for the larger sized websites, even small sites can really benefit from having them in place. And in fact, in many cases, a sitemap is actually one of the simplest ways to improve the likelihood of your pages getting discovered and indexed.
Now, this certainly doesn’t replace good internal linking, but it does act as a clear guide for the search engines. And because of that, this is a lesson you’re definitely not going to want to skip.
An XML sitemap is a file that’s hosted on your website that provides search engines with a structured list of your pages. It tells the search engines where your pages are located and when they were last updated. And this helps them to understand the overall structure of your site and decide which pages to crawl and recheck.
They’re designed specifically for the search engines and they’re not really intended for users. In fact, most visitors will never see an XML sitemap unless they go looking for it intentionally.
XML sitemaps help search engines to discover pages that may not be easy to find through links alone. This includes new pages and pages that don’t yet have any links pointing to them, which are often referred to as orphan pages.
Sitemaps also improve crawling efficiency since search engines don’t have unlimited resources, and anything that helps them to understand your site more clearly is going to make their job a lot easier.
The sitemap also includes information about when a page was last updated. And so when the search engines notice a change in the file, it can encourage them to recrawl those pages and refresh what they have stored in their index.
The type of sitemap you use depends on how your website is built and how often it changes. Now, a static sitemap is a file that you have to update manually. So, every time you add, remove, or change a page, you’re going to need to update this file yourself and then re-upload that back to the server.
Whereas, with a dynamic sitemap, this is generated automatically using a plug-in or it may even be built into your website system. This is one that updates itself as your site changes without you having to do any manual work at all.
So you can add, remove or change pages and it will be automatically updated in your XML sitemap. As you can imagine, static sitemaps can be quite time consuming, especially on larger websites where pages get changed quite often. They can also create indexation problems too if you forget to update them.
However, there are some scenarios they can still be very useful. For example, on small websites that rarely change or for very specific sections of websites where automation isn’t practical.
Now, if your website is built using a CMS like WordPress, Shopify, Drupal, Magento, and so on, there’s almost certainly going to be a plug-in that you can use to create a dynamic XML sitemap and 99% of the time that’s what I’d recommend as it’s going to automate the process and stop you from having to put in any manual effort.
Once you have your sitemap created, the next step is to tell the search engines where it is, and that’s what we’re going to look at in this part of the lesson.
Let’s first take a look at adding your sitemap to Google. Now, in a previous lesson, we set up Google Search Console. And if you haven’t done that yet, make sure you go back through that earlier lesson and get Google Search Console set up for your website. Once you’ve got that, you’ll see a report that looks like this. And you’ll be able to click on sitemaps on the left-hand side.
Now you’ll see here it says add a new sitemap and it’s got the domain name followed by the forward slash and it says enter sitemap URL. That means you want to enter everything after the forward slash in your URL. So in my example here, that I’ve got on my website is going to be at sitemap_index.xml.
So I’m going to paste that into there and then I would just click on submit and that would be all I need to do to get my sitemap into Google.
Bing also have their own platform that allows you to see how your website is performing inside of their search engine and it’s called Bing Webmaster Tools. Now it’s a very similar system to Google Search Console and you can actually import your website in from Google Search Console without having to do any extra validation.
All you need to do is head over to bing.com/webmasters. And when you get there you’ll see a page that looks like this. On the left, you’ll see it says here that you can import your sites from GSC, Google Search Console.
So click on import and then you’ll want to click on continue and then it will give you the option as to which Google account you want to sign in using. You’ll want to make sure that you use the exact same Google account that you used when you set up Google Search Console.
You’ll then get this message come up. You’ll want to click on continue and then continue again. From here, it will show you a list of all the different websites that you have inside of your Google Search Console and you just click on import. And that’s all you need to do. And as long as you’ve already submitted your sitemap to Google Search Console, it will automatically pull that over inside of Bing Webmaster Tools.
The robots.txt file is the very first thing a search engine looks at when it visits your site. Before Google or Bing crawls a single page, it checks the robots.txt to see what it’s allowed to access and where the important things are.
By placing your sitemap inside of that file, you are making it easier for search engines to discover and understand your pages right from the start. It also means that the smaller search engines can find your sitemap, too. And it only takes a couple of minutes to add.
So, you want to come into your FTP client and look in the root directory of your server. And you want to find the file name robots.txt. You then want to edit the robots.txt. And when you edit that, you’ll likely see that it looks something like this. Now, what you want to do is add this line: capital S sitemap colon space and then the full URL to your sitemap.
Once you’ve added that in, save it back to the server. You’ll then want to head back to your web browser and go to your robots.txt file on your website to just make sure that your site map has been included in your robots.txt.
You’ll finally want to just take this URL right here and paste that into your browser just to make sure that you have put the correct URL in.
HTML sitemaps are quite different from XML sitemaps. Instead of it being a hidden file that’s designed just for search engines, a HTML sitemap is a normal web page that real people can actually visit. Just like an XML sitemap, it contains links to all the pages on your site, but it’s usually linked to from the footer of your website. This means that users can click on it and use it to navigate their way around your website.
Now, from a search engine point of view, it’s really useful because it helps to reinforce your site structure and makes it easier for crawlers to move through your pages. Similar to XML sitemaps, HTML sitemaps can be either static or dynamic. If your site is built on a CMS, you can usually find a plug-in or you can write some custom code yourself to handle this for you.
However, if your website is static, you will need to create a static HTML sitemap. And this means that you’re going to need to update it every time a page is added or removed.
Now, it’s important to understand that this is not a choice between one or the other. XML sitemaps and HTML sitemaps serve different purposes, and there’s really no downside to having both. And when you do use them together, you make it much easier for search engines and users to understand and navigate your website.
So, now that we’ve covered the main ways that pages are found and prepared for search results, it’s time to move on to the next lesson, where we’ll learn about website structure and how organizing your pages properly can help both users and search engines to understand your website more clearly.