Index Drupal 9 pages with IndexNow using Simple XML sitemap

15 Feb 2022

Bing 410 Gone

Microsoft recently killed their search engine's public API responsible for accepting sitemap ping requests (those where you let search engines know your XML sitemap's content has changed). They did so completely unannounced leading to logs filling up and users unsurprisingly assuming their sitemap submitting code was somehow at fault. It wasn't.

It became apparent that this was a step for Microsoft towards switching to the IndexNow protocol instead.

IndexNow supplements XML sitemaps

With IndexNow you now can quickly notify all participating search engines (Bing, Yandex) about a change that happened on your page eliminating the need to wait for them to come and scan your sitemap. The benefits of this approach are

  • Instant submission of changes including creating, deleting and updating of content
  • An alleged greener approach to indexing content as sitemap scanning requests get deprioritized
  • Only one search engine needs to be notified and it will notify the others for you

Still, this new approach is more of a supplement than a revolution:

Can I submit all URLs for my site?

Use IndexNow to submit only URLs having changed (added, updated, or deleted) recently, including all URLs if all URLs have been changed recently. Use sitemaps to inform search engines about all your URLs. Search engines will visit sitemaps every few days.

I have a sitemap, do I need IndexNow?

Yes, when sitemaps are an easy way for webmasters to inform search engines about all pages on their sites that are available for crawling, sitemaps are visited by Search Engines infrequently. With IndexNow, webmasters ''don't'' have to wait for search engines to discover and crawl sitemaps but can directly notify search engines of new content.

Source: https://www.indexnow.org/faq

Simple XML sitemap now supports IndexNow

To further their cause Microsoft recently released the IndexNow plugin for Wordpress. Well we've been working hard to bring IndexNow to Drupal. To use it, install and enable the latest version of Simple XML sitemap (simple_sitemap) and its Engines submodule:

  1. composer require 'drupal/simple_sitemap:^4'
  2. drush en simple_sitemap simple_sitemap_engines

If you are using the older 8.x-2.x or 8.x-3.x, don't worry, the module will upgrade.

If you had Engines submodule submit the default XML sitemap to Bing, the update hooks will make sure to enable submitting the corresponding entities to IndexNow on every entity form save (don't worry, given the right permission, this can be overridden before submitting the form).

Set up IndexNow

You will have to generate a new verification key on your production sites - just head to the status page (admin/reports/status), or directly to admin/config/search/simplesitemap/engines/settings. The key can be stored in Drupal state or, better yet, in settings.php. If you don't want your non-production environments to submit changes, just don't generate a key.

The new IndexNow functionality is well integrated into Simple XML sitemap and so all content entities can be indexed and indexation settings can be found in familiar places. To configure which entity types are to be indexed by IndexNow, head to admin/config/search/simplesitemap/entities.

The module supports sending changes on form sumbissions as well as on all entity save operations. The preferred IndexNow engine can be chosen or a random one will be picked on each submission. Just head to admin/config/search/simplesitemap/engines/settings to set your preferences.

Ways to contribute

If you would like support the project, feel free to contribute code, bug reports and translations. I am also keen on reading comments that were not written by bots. ;)

All info about the project

Comments

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang target> <em> <strong> <cite> <blockquote cite> <pre> <ul type> <ol start type> <li> <dl> <dt> <dd> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.

Get a quote in 24 hours

Wether a huge commerce system, or a small business website, we will quote the project within 24h of you pressing the following button: Get quote