On our Production Platform we have Heap troubles which obliges us to restart our cluster every 10days~
(Heap of each nodes increase to 100%, so timeout on all cluster etc...)
data : 33TB, 700indx replicat 1
We don't know why we have this heap trouble (nothing very relevant on logs...)
We have 3 master nodes and 6 datanodes on elastic 5.5.0, configured with 31gb Heap.
As our servers have 256gb RAM, I think we can configure ES to have 2 datanodes on each datanodes servers, even if i see it's not recommended on PRODUCTION platform (with configuration node.max_local_storage_nodes: 2).
1st : is it a good thing to do ? What are the recommended configurations for that (i didn't see anything on elastic documentation on that point)
2: How work shards allocation on that case ? (Are we sure there was not primary shards and secondary on 2 nodes of the same host/server ?
FI: I test a configuration on vm, 2 systemd with different conf path /opt/application/elasticsearch/current/config/node01 and node02 with param max local storage but i'm not sure it will be the best things to do on production