Index Drupal 9 pages with IndexNow using Simple XML sitemap

15 Febr. 2022

Bing 410 Gone

Microsoft recently killed their search engine's public API responsible for accepting sitemap ping requests (those where you let search engines know your XML sitemap's content has changed). They did so completely unannounced leading to logs filling up and users unsurprisingly assuming their sitemap submitting code was somehow at fault. It wasn't.

It became apparent that this was a step for Microsoft towards switching to the IndexNow protocol instead.

IndexNow supplements XML sitemaps

With IndexNow you now can quickly notify all participating search engines (Bing, Yandex) about a change that happened on your page eliminating the need to wait for them to come and scan your sitemap. The benefits of this approach are

  • Instant submission of changes including creating, deleting and updating of content
  • An alleged greener approach to indexing content as sitemap scanning requests get deprioritized
  • Only one search engine needs to be notified and it will notify the others for you

Still, this new approach is more of a supplement than a revolution:

Can I submit all URLs for my site?

Use IndexNow to submit only URLs having changed (added, updated, or deleted) recently, including all URLs if all URLs have been changed recently. Use sitemaps to inform search engines about all your URLs. Search engines will visit sitemaps every few days.

I have a sitemap, do I need IndexNow?

Yes, when sitemaps are an easy way for webmasters to inform search engines about all pages on their sites that are available for crawling, sitemaps are visited by Search Engines infrequently. With IndexNow, webmasters ''don't'' have to wait for search engines to discover and crawl sitemaps but can directly notify search engines of new content.

Source: https://www.indexnow.org/faq

Simple XML sitemap now supports IndexNow

To further their cause Microsoft recently released the IndexNow plugin for Wordpress. Well we've been working hard to bring IndexNow to Drupal. To use it, install and enable the latest version of Simple XML sitemap (simple_sitemap) and its Engines submodule:

  1. composer require 'drupal/simple_sitemap:^4'
  2. drush en simple_sitemap simple_sitemap_engines

If you are using the older 8.x-2.x or 8.x-3.x, don't worry, the module will upgrade.

If you had Engines submodule submit the default XML sitemap to Bing, the update hooks will make sure to enable submitting the corresponding entities to IndexNow on every entity form save (don't worry, given the right permission, this can be overridden before submitting the form).

Set up IndexNow

You will have to generate a new verification key on your production sites - just head to the status page (admin/reports/status), or directly to admin/config/search/simplesitemap/engines/settings. The key can be stored in Drupal state or, better yet, in settings.php. If you don't want your non-production environments to submit changes, just don't generate a key.

The new IndexNow functionality is well integrated into Simple XML sitemap and so all content entities can be indexed and indexation settings can be found in familiar places. To configure which entity types are to be indexed by IndexNow, head to admin/config/search/simplesitemap/entities.

The module supports sending changes on form sumbissions as well as on all entity save operations. The preferred IndexNow engine can be chosen or a random one will be picked on each submission. Just head to admin/config/search/simplesitemap/engines/settings to set your preferences.

Ways to contribute

If you would like support the project, feel free to contribute code, bug reports and translations. I am also keen on reading comments that were not written by bots. ;)

All info about the project

Comments

Hi

Sorry if I'm being thick.

I've read several sets of documentation for this and they all say "The key can be stored in Drupal state or, better yet, in settings.php. " or similar.

Before I go and mess up the complex D10 settings.php, I'd rather make sure I know exactly what I'm doing!

Is there a specific location?

Is there a specific expression?

Or do I just plonk the key somewhere in the file?

Thanks.

Neuen Kommentar hinzufügen

Der Inhalt dieses Feldes wird nicht öffentlich zugänglich angezeigt.

Restricted HTML

  • Erlaubte HTML-Tags: <a href hreflang target> <em> <strong> <cite> <blockquote cite> <pre> <ul type> <ol start type> <li> <dl> <dt> <dd> <h4 id> <h5 id> <h6 id>
  • Zeilenumbrüche und Absätze werden automatisch erzeugt.
  • Website- und E-Mail-Adressen werden automatisch in Links umgewandelt.

Angebot innerhalb von 24 Stunden

Ob ein großes kommerzielles System, oder eine kleine Business Seite, wir schicken ein Angebot ab innerhalb von 24 Stunden nachdem Sie diese Taste drücken: Angebot anfordern