Parallel crawler execution

Elastic Search
1

I would like to create a crawler per domain. This would allow crawling each domain independently and running them at different times.

However, it doesn't seem possible. If running the crawler consumes a lot of resources, is there a way to allocate more resources to it ?

2

Have you tried the crawler.workers.pool_size.limit config?

From the reference config file:

# The number of parallel crawls allowed per instance of Enterprise Search.
# By default, it is set to 2x the number of available logical CPU cores.
# Note: On Intel CPUs, the default value is 4x the number of physical CPU cores
# due to hyper-threading (https://en.wikipedia.org/wiki/Hyper-threading).
#
#crawler.workers.pool_size.limit: N
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.