Elasticsearch crawler - Usage of "lastmod" properties in sitempas

Elastic Search
1

Hi,

I am looking for documentation about how the Elasticsearch Crawler is handling the "lastmod" properties in website sitemaps. But so far, I am not able to find IF ES is actually taking the lastmod in consideration and actually crawl/update only the page is a lastmod newer than the previous crawl.

Thanks,

David

2

Hi!
I don’t think the Crawler is taking the lastmod parameter into account. Also:
crawler/docs/features/CRAWLER_DIRECTIVES.md at d6c749b5888f6e3f53b3a798ee655cea341f447d · elastic/crawler · GitHub states:

The web crawler does not process optional metadata defined by the standard.
The web crawler extracts a list of URLs from each sitemap and ignores all other information.