Does Elastic Web Crawler supports noindex and nofollow directive

Hi @Sean_Story,

no worries. We can investigate a bit deeper. But what is the expected behavior for the Elastic Web Crawler from your point of view? In which case pages are deleted?

case 1: page with index meta tag is in index -> page switches to noindex -> full re-crawl -> page deleted from index: yes vs. no

case 2: page with index meta tag is in index -> page switches to noindex -> partial crawl -> page deleted from index: yes vs. no

case 3: page with index meta tag is in index -> page is deleted -> HTTP 404 -> full re-crawl -> page deleted from index: yes vs. no

case 4: page with index meta tag is in index -> page is deleted -> HTTP 404 -> partial crawl -> page deleted from index: yes vs. no

Other cases for deleting?

We only need to understand and can the investigate and decide how to delete or force deletion of pages from our indices.

Best regards

Sebastian