Do you use monitoring tools for watching the cluster nodes?
So you can find out how the resource usage is developing until you reach 25
mio. I predict you will notice the cluster entering big segment merge phase
plus the search load from your scan/scroll requests. Try to streamline
segment merging by either throttling or reducing the segment maximum size
to load (default is 5G).
You should try using a smaller value for setSize(), maybe 200 instead of
2000, to let the scan/scroll generate more handy bulk request sizes.
The life time for a scroll request is very high, 2 minutes. During this
time the server must keep found docs in memory and this can easily pile up.
I would reduce it to 30 seconds or so. This will save resources on the
cluster node, but it must be balanced with the setSize() param to avoid
search timeouts.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.