Can't index an intranet domain website - Enterprise Search 7.15

Elastic Search
1

Team,

I'm trying to index a company's intranet domain but only one document is indexed. I have my setup using docker-compose and have these services (Elasticsearch, Enterprise-Search, and Kibana).

A thing to note that when I cURL to the domain from a VM that is on the same network the result is 200 OK but there is no complete HTML page instead it contains more JavaScript code.

Any direction what would be the problem ? My guess that it may be restricted from the company to crawl the contents or it may be related to the authentication ?

Not sure how to proceed or what settings needs to be adjusted to index the data.
Kind Regards.

2

Hey @baseely,

A thing to note that when I cURL to the domain from a VM that is on the same network the result is 200 OK but there is no complete HTML page instead it contains more JavaScript code.

This sounds like it's probably the cause. The App Search crawler will analyze and index the HTML response for a given page, but it wont execute Javascript or other dynamic content when crawling a domain. That functionality is absolutely on the future roadmap.

Ross

3

Thanks Ross for your reply.

1 Like