We are trying to index large files in elastic search. Elastic heap memory consumption shoots up while indexing.
Basic observation is that while indexing memory can go upto 30x and post indexing it remains at around 10x.
For eg: while indexing 600MB file, memory usage is around 6GB and it can shoot upto 18 GB.
while indexing 1GB file, memory usage is around 10GB and it can shoot upto 30 GB.
Is this expected behaviour. Is there a way we can reduce the memory footprint.
Any help with this query is highly appreciated.
This behavior is seen in both ES 2.4 and ES 5.5 versions
2 data nodes, 1 dedicated master node.
ES Heap memory - 31 GB in all nodes