Hi Team,
We recently upgraded our Elasticsearch cluster from 7.6.2 to 7.17.1, and after a few days, it seems working very well, but when we doing some heavy indexing job, the cluster start to crash in an randomly way.
The indexing and search throughputs are normal, and the data structure are unchanged recently, after fallback to version 7.6.2, the cluster became stable, so we can confirm it is only happend in the version of 7.17.1, we also noticed 7.17.2 was released today, but didn't have time to test it yet.
We touched the dump file, here is the result:
It seems the code is try allocating memory for posing arrays of some term field, and obviously the memory is not enough and then crashed OOM.
The metrics of index writer
shows that the memory when it crash was out of the default limit, which is
10%, should be less then 2.4GB, as the total heap size is 24GB, but jumps to 7.5GB, which is unusual, the circuit breaker seems not works as expected.
Is this is a already know issue?
Appreciate for any help, Thanks.
/Medcl