After upgrading a cluster from 2.3 to 5.5, I'm noticing an increase in total memory. I don't have enough data yet, but it also looks to grow (slowly) unbounded until OOM. I've read other accounts (not strictly elasticsearch related) how this behavior can look like a memory leak but really it's more like an optimization ("PSST! Don't really free this memory because I'm going to need it right back, ok?").
Elasticsearch is running inside a container. It has 30G heap. The host system has 128G. Using openjdk 1.8.0_131.
1st question, how can I restrict the amount of off-heap usage in order to avoid OOM's? I tried setting
-XX:MaxDirectMemorySize and also tried
MALLOC_ARENA_MAX=2. The malloc arena experiment didn't yield much different behavior: total resident memory quickly grows 80+G. Setting MaxDirectMemorySize=32G improved the situation slightly: resident memory reached around 64G but still continued to climb, albeit at a much slower rate.
2nd question, how can I better understand what is driving the off-heap usage? An increase in this usage from 2.x to 5.x was expected since doc_values are the default. Any tips or references are greatly appreciated. Thanks!