Bing 410 Gone
Microsoft recently killed their search engine's public API responsible for accepting sitemap ping requests (those where you let search engines know your XML sitemap's content has changed). They did so completely unannounced leading to logs filling up and users unsurprisingly assuming their sitemap submitting code was somehow at fault. It wasn't.
It became apparent that this was a step for Microsoft towards switching to the IndexNow protocol instead.
IndexNow supplements XML sitemaps
With IndexNow you now can quickly notify all participating search engines (Bing, Yandex) about a change that happened on your page eliminating the need to wait for them to come and scan your sitemap. The benefits of this approach are
- Instant submission of changes including creating, deleting and updating of content
- An alleged greener approach to indexing content as sitemap scanning requests get deprioritized
- Only one search engine needs to be notified and it will notify the others for you
Still, this new approach is more of a supplement than a revolution:
Can I submit all URLs for my site?
Use IndexNow to submit only URLs having changed (added, updated, or deleted) recently, including all URLs if all URLs have been changed recently. Use sitemaps to inform search engines about all your URLs. Search engines will visit sitemaps every few days.
I have a sitemap, do I need IndexNow?
Yes, when sitemaps are an easy way for webmasters to inform search engines about all pages on their sites that are available for crawling, sitemaps are visited by Search Engines infrequently. With IndexNow, webmasters ''don't'' have to wait for search engines to discover and crawl sitemaps but can directly notify search engines of new content.
Simple XML sitemap now supports IndexNow
To further their cause Microsoft recently released the IndexNow plugin for Wordpress. Well we've been working hard to bring IndexNow to Drupal. To use it, install and enable the latest version of Simple XML sitemap (simple_sitemap) and its Engines submodule:
- composer require 'drupal/simple_sitemap:^4'
- drush en simple_sitemap simple_sitemap_engines
If you are using the older 8.x-2.x or 8.x-3.x, don't worry, the module will upgrade.
If you had Engines submodule submit the default XML sitemap to Bing, the update hooks will make sure to enable submitting the corresponding entities to IndexNow on every entity form save (don't worry, given the right permission, this can be overridden before submitting the form).
Set up IndexNow
You will have to generate a new verification key on your production sites - just head to the status page (admin/reports/status), or directly to admin/config/search/simplesitemap/engines/settings. The key can be stored in Drupal state or, better yet, in settings.php. If you don't want your non-production environments to submit changes, just don't generate a key.
The new IndexNow functionality is well integrated into Simple XML sitemap and so all content entities can be indexed and indexation settings can be found in familiar places. To configure which entity types are to be indexed by IndexNow, head to admin/config/search/simplesitemap/entities.
The module supports sending changes on form sumbissions as well as on all entity save operations. The preferred IndexNow engine can be chosen or a random one will be picked on each submission. Just head to admin/config/search/simplesitemap/engines/settings to set your preferences.
Ways to contribute
If you would like support the project, feel free to contribute code, bug reports and translations. I am also keen on reading comments that were not written by bots. ;)