I am setting up App Search and need to crawl some internal content. I found this post that says the feature is coming in the next minor patch, but it wasn't clear which version that will be. Will it be a patch based off 7.14 or does 7.14 already include it? If 7.14 includes the new feature how do we toggle it?
Hello, Lane,
Thank you for trying out our new web crawler!
You can check your enterprise-search.yml file for the following two settings (those have been around for quite some time now):
# -----------------------------
# Crawler DNS Security Controls
# -----------------------------
#
# WARNING: The settings in this section could make your deployment vulnerable to
# SSRF attacks (especially in cloud environments) from the owners of any domains
# you crawl. Do not enable any of the settings here unless you fully control DNS
# domains you access with the crawler.
#
# See https://owasp.org/www-community/attacks/Server_Side_Request_Forgery for
# more details on the SSRF attack and the risks associated with it.
#
# Allow crawler to access the localhost (127.0.0.0/8 IP namespace)
#
#crawler.security.dns.allow_loopback_access: false
#
# Allow crawler to access the private IP space: link-local, network-local addresses, etc
# (see https://en.wikipedia.org/wiki/Reserved_IP_addresses#IPv4 for more details)
#
#crawler.security.dns.allow_private_networks_access: false
Those should allow you to control where the crawler is allowed to go.
Please note: Those are not available on Elastic Cloud for security reasons.
1 Like