Hi all,
I set up an index containing 500.000 with million nested documents and I
run big queries with a lot of aggregation. As expected this is using a LOT
of memory and I easily have an OutOfMemory exception if a user perform a
query for too many documents
So I configured elasticsearch with the folloting:
indices.fielddata.cache.size: 20%
indices.fielddata.breaker.limit: 20%
indices.fielddata.breaker.overhead: 1.2
indices.fielddata.cache.expiry: 1m
indices.cache.filter.size: 20%
indices.cache.filter.expire: 1m
(I've two nodes with one index running on m3.xlarge EC2 instances with 5
shards per node HEAP size is 8G (half of total RAM)).
Now I've well this beautiful error message when a user is performing a big
query:
CircuitBreakingException[Data too large, data would be larger than limit of
[2570007347] bytes
If I check the nodes, the field cache reached its maximum size (1.6G) as
expected. Then I can easily returned an error message to the user telling
him to downsize the scope of its query and it should be fine.
But... This will never let me run a query anymore!
The cache is filled up for ever and all query, even basic small search
(without aggregation) will be responded with the CircuitBreakingException.
The only way I found so far is manually clearing the cache. Of course this
is not an acceptable solution for production.
*How can I make sure this cache will never be continuously filled up? *
How can I automatically remove from the cache unused data?
Does indices.fielddata.cache.expiry config param actually work?
Thanks all!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/72f68e09-c670-494c-af77-328e37e4aa0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.