We recently upgraded our cluster from ES 2.4 to ES 5.2. The upgrade went smoothly but upon starting everything back up we noticed a lot of instability particularly with garbage collecting. What used to be a very stable and reliable cluster is now dropping nodes due to heap and cpu spikes. Usage, indexes, docs, shards, hardware are all the same.
We are currently trying to get to the bottom of why we are suddenly so unstable and why our heap usage seems out of control but are not having much luck. So snippets that might be useful
- Avg Indexing rate 1k docs/sec (can get to 3k/s)
- refresh rate=30s to accommodate indexing
- Some heavy aggregations but nothing that has caused problems in the past
- java version "1.8.0_31"
Was there a change in ES 5 that could cause heavy aggregations to suddenly become more of a burden? Or was there a change that could have caused large amounts of indexing to cause heap and cpu spikes?
Any suggestions at all would be greatly appreciated! Let me know if more information is needed!