I am new to elk stack. We already have a ES 2.4 cluster running and are trying to upgrade to 5.1.1 version. I am feeding live data to old(2.4.x) and new(5.1.1) cluster from logstash(5.x). The data is now in sync between these two cluster but old cluster's index size is almost double when compared to new cluster's index size. Below is an example:
ES Version health status index pri rep docs.count docs.deleted store.size pri.store.size
2.4.x Old_Cluster green open index1-08 5 1 6520824 0 5.3gb 2.6gb
5.1.1 New_Cluster green open index2-08 5 1 6520824 0 9.3gb 4.6gb
I have made modification to the mappings thinking this might change the index size. Following is my current mapping on 5.1 cluster: https://gist.github.com/anonymous/563f88950342f7d910a579995cad1fe6
This didnt make any change to the index size as shown below:
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open new_cluster_index 5 1 6425708 0 9.2gb 4.6gb
I am not sure but i think this behaviour is resulting in high search response times in kibana when compared to the old kibana running with ES 2.4 version.
Can anyone provide any suggestions on this? I am not sure where the problem is occurring in kibana or elasticsearch. Is this something expected when coming to ES 5.1.1 cluster?