Throttling fixed bitset cache

(Jérôme Mainaud) #1


Is there any way to throttle fixed bitset cache (in 1.7 or at least in 2.x)?
I have a client cluster whose 8 GB heap memory is saturated by fixed bitset cache after five minutes.

The cluster avoid OutOfMemoryException by setting index.load_fixed_bitset_filters_eagerly to false and clearing cache every 5 minutes, which is not a durable situation.

Other caches (cache field and cache filter) are OK.

I know that the principal reason is due to massive usage of nested fields in filters and sort using neasted_filter as in and that the best solution is to migrate to Elasticsearch 2.2.

But I would like to be able to throttle this cache. First as a temporary solution, the time to do the migration since the latter requires time and eviction on fixed bitset cache should be better than periodically clearing all caches. Then, because I don't want to prevent node crashes in case migration to 2.2 would not fix the problem.

Thank you.

(Adrien Grand) #2

There was a bug that caused this cache to duplicate its content, maybe you are affected?

(Jérôme Mainaud) #3

That might be a cause since their real version is 1.7.3.

By the way I support your idea to read fixed bitset from the disk.
It would be a good idea.

Thank you, Adrien.

(Adrien Grand) #4

You're welcome Jérôme!

(system) #5