Hi all,
I have an elasticsearch index containing around 65 million documents . Two days before, I updated the mapping of the index by adding 8 more new fields some of which are nested. After that around 15 million of the documents were updated along with these fields using bulk api . Python's elasticsearch client was used and the update was done in batches of 500. After the update, the latency of elastic queries started to increase. A query that used to take around 1-2 seconds now started taking more than 5 seconds.The health of all the different nodes are fine and I couldnt find any unnatural spikes in any of the elastic metrics. The cpu utilisation is also well within limits.
I am using elastic search version 2.4.4 with 3 nodes and 5 shards with 1 replica. Each machine has 6 cores,12 gb ram and 190gb disk space. The size of the index is 103Gi .The heap size of the cluster is 8gb
Ordinarily the number of updates done to elastic will be less than a million and they dont cause any issues. I had faced the same issue once before where after doing a large number of updates the latency started to increase and then it got automatically better after a couple of days.
I would like to know the reason for this increase in latency and measures I could take to prevent this from happening again