An issue with the same subject was raised in April 2020. . orhantoy Elastic Team Member advised "Yes, that's the current, default behavior and it will become configurable in the next minor release." Has this become configurable?

Web Crawler Failed HTTP request: Unable to request “< domain >” because it resolved to only private/invalid addresses v2

ross.bell (Ross Bell) November 1, 2021, 4:56pm 3

Topic		Replies	Views
Web Crawler Failed HTTP request: Unable to request "< domain >" because it resolved to only private/invalid addresses Elastic Search elastic-app-search	4	1121	May 18, 2021
Add a domain to get started - validate domain - failed Elastic Search	2	662	November 4, 2022
Web crawler DNS name resolution failed while validating domain Elastic Search elastic-app-search	2	302	November 1, 2023
Crawler always fails when there are several domains Elastic Search elastic-app-search	3	430	December 20, 2021
Web Crawler Setup: "Content Verification" for Domain fails Elastic Search elastic-app-search	5	652	February 10, 2022