Understanding and Creating Sitemaps for SEO

Open Table of contents

What is sitemaps
Who needs sitemaps
How to create a sitemap
Example code for automating sitemap generation using Laravel

What is sitemaps

A sitemap is a file listing all the pages on a website. It serves as a root map for search engines to discover and index the contents of a website. It is crucial for a improving a websites’s visibility in search engine.

Who needs sitemaps

If you have a relatively small website and your pages are properly linked, search engine crawler bots can usually discover your content so you don’t have to worry about sitemaps.

But if your site is really large or your pages are isolated and not linked to each other or your site is new or has a lot of changing contents (like a news website) it’ll be a wise idea to add a sitemap.

Having a sitemap is not a 100% guarantee that your pages will be crawled and indexed by the search engines.

In most cases the site will benefit from it, and there’s no disadvantage in having one.

Sitemaps don’t replace the normal crawling and if you don’t add a URL in the sitemap, the search engine could still index the contents of that URL.

How to create a sitemap

Ideally the system running your website should generate a sitemap for you automatically, example: you can find a Wordpress plugin or a Drupal extension if you are using a CMS system. If your website is running on a custom server, you’ll have to manage the sitemap creation yourselves. Most popular frameworks will have convenient packages that helps you generate sitemaps.

You’ll have to keep in mind that there are limits to number of URLs a sitemap file can include and maximum size a sitemap file can have. So you can create multiple sitemap files to get around this and create an index sitemap file to combine them all

Example code for automating sitemap generation using Laravel

Here I’m using the a package called spatie/laravel-sitemap for generating sitemaps

Install the spatie/laravel-sitemap package

composer require spatie/laravel-sitemap

If you’d like to customize the default options of this package, you can publish the configuration file with this command
```
php artisan vendor:publish --provider="Spatie\Sitemap\SitemapServiceProvider" --tag=sitemap-config
```
Create a new job to handle sitemap creation, run the following artisan command to create a new Job
```
php artisan make:job SitemapJob
```

You can use a Job like the following to handle the sitemap creation

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Spatie\Sitemap\Sitemap;
use Spatie\Sitemap\Tags\Url;

class SitemapJob implements ShouldQueue
{

    private array $paths;
    private  string $sitemapPath;
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct($paths, $sitemapPath)
    {
        $this->paths = $paths;
        $this->sitemapPath = $sitemapPath;
        $this->onQueue('sitemap_generation');
    }

    public function handle()
    {
        $sitemap = Sitemap::create();
        foreach ($this->paths as $path){
            $sitemap->add(Url::create('https://yourdomain.com'.'/'.$path));
        }
        $sitemap->writeToFile($this->sitemapPath);
    }
}

Now to automate the sitemap generation task we can create a custom artisan command, run the following command to scaffold a custom command
```
php artisan make:command GenerateSitemap
```

You can use a command like the following to automate the sitemap generation

<?php

namespace App\Console\Commands;

use App\Jobs\SitemapJob;
use Illuminate\Console\Command;
use Spatie\Sitemap\SitemapIndex;

class GenerateSitemap extends Command
{
    const PER_PAGE = 500;

    protected $signature = 'sitemap:generate';

    protected $description = 'Generate sitemap for the website';

    public function __construct()
    {
        parent::__construct();
    }

    public function handle(): int
    {
        $sitemap_index = SitemapIndex::create();

        $sitemap_url = config('sitemap.base_url') . config('sitemap.static_storage_path');
        $sitemap_index->add($sitemap_url);
        SitemapJob::dispatch(config('sitemap.static_paths'), public_path(config('sitemap.static_storage_path')));

        $dynamicPages = []; // You can populate this list from your database (take all the dynamic page paths that you want to add in the sitemaps)
        $i = 0;
        $dynamicPages->chunk(self::PER_PAGE, function ($pages) use ($sitemap_index, &$i) {
            $filepath = "sitemaps/page_sitemap_{$i++}.xml";
            $sitemap_url = config('sitemap.base_url') . $filepath;

            $sitemap_index->add($sitemap_url);
            SitemapJob::dispatch($pages->pluck('page_name')->toArray(), public_path($filepath));  // this line assumes that your `dynamicPages` have a property called 'page_name'
        });

        $sitemap_index->writeToFile(public_path('sitemap.xml'));
        return 0;
    }
}

Here all the configs mentioned can be added in config/sitemap.php which will be added when publishing the package configuration file with the given command above.

Now if you want to further automate running this command, Laravel has got your back. You can make use of laravel’s task scheduling system to run this command on a given intervals. For example you can add the following line of code in your schedule function inside the app/Console/Kernel.php to run the sitemap generation command at midnight on every Sundays
```
$schedule->command('sitemap:generate')->sundays()->at('00:00');
```