I have a 12 node cluster + 3 masters, running ES 2.3.1, with 5000 shards cluster wide, about 410 shards per node, roughly 250 billion small documents.
Shards are roughly 8GB in size, well balanced (small variance of shard size in all the indices).
Shards are also optimized to 6 segments.
Data nodes are with 64GB main memory, with 31GB allocated to ES.
Currently, when the cluster is idle, heap consumption in every data node is around 20GB (out of 31GB).
I believe almost all the memory is used for the segment in-memory needs, because query and field data caches are empty and according to _segments, every shard is using about 50MB of memory_in_bytes, so 50MB * 410 = 20GB.
doc_values are disabled for all indices to save space, and the cluster is used for non aggregation queries only.
Is there a way to control memory_in_bytes of lucene segments?
I dug up some old posts about termInfosIndexDivisor, but they seem to be irrelevant today....
Thanks in advance