we have an issue with ES 5.2 (3 node cluster) where the index size keeps growing disproportionately. We run some background job everyday which indexes 5million existing cases in 3 hr window. we have refresh interval set to 1s as required by business. The update in documents is marginal. like 1% of total index size.
So far this has been working well with our old ES 1.3.4. we run same job everyday on 1.3.4 as well. During the run the index size grows and falls back to expected size due to segment merges. But in 5.2 this size never drops. we can see the merges are happening but for some reason the index size just grows.
Example:
ES 1.3.4
500k documents
initial index size: 100GB after a job run 125gb, after sometime reduces back to 112gb. (2gb increase due new info on existing documents)
ES 5.2
500k documents
initial index size: 100GB after job run 125gb, size drop back by hardly 3-5gb. This means the index in 5.2 grows by 22gb everyday vs 2gb in 1.3.
Cluster spec:
3 node, each node 2.5tb storage, 16gb heap, 2 master nodes, index has 5 primary shards, with replication factor 1, memory and OS swap disabled, refresh_interval 1s. Running on 64 bit linux, jdk 1.8_0_20_64bit
Also, have tried below settings to tune merging:
"index.merge.scheduler.max_merge_count": 100,
"index.merge.scheduler.max_thread_count":4,
"index.merge.policy.max_merged_segment": "12gb",
"index.merge.policy.reclaim_deletes_weight":3.0,
"index.merge.policy.max_merge_at_once":10,
"index.merge.policy.segments_per_tier":10,
"index.merge.policy.floor_segment":"5mb"
Also tried _forceMerge. which merges segments as expected freeing up deleted docs space.
but results from "myindex/ _segments" seems odd. Adding all segments "size_in_bytes" doesnt match the size of index??
i.e.
If I have 5 segments then their sum of "size_in_bytes" is 10gb vs, the shard size under "_cat/shards?v" shows 70gb? The docs count matches as expected though.
Also, checked the linux "top" command output to see the memory is locked for each node. it shows node1 has 146gb, node2 67gb, node3 has 35gb.
When we restart ES 5.2 cluster, the index size returns to expected 100gb initial state but then keeps growing with daily indexing run.
Have checked for possible doc_values, we are using same fields in 1.3.4 and 5.2.
Pls help.