Update with info I should have included!
ES version 6.8.12.
Three nodes in the cluster running on Linux. Two nodes have SSDs and one has rotating disks -- this was added for resilience -- all api calls go to the two with fast disk which are "hot" the other is "warm".
I have not found anything in the logs that seemed relevant.
Whenever I restart a node in the cluster it loses all its shards which are then reallocated from backups on other nodes. This is in spite of me having tried to turn of allocations ( "cluster.routing.allocation.enable": "primaries"
) or set the reallocation wait period to enough time for the node to come back online(
index.unassigned.node_left.delayed_timeout": "10m"
).
Any suggestions as to what I am missing ?