Ok to add safe-to-evict annotation to Elasticsearch/Kibana pods for k8s cluster autoscaling?

[jketcham (Jack Ketcham)]

I'm running ECK in Google Kubernetes Engine, and I've run into instances where the Cluster Autoscaler will choose not to remove an ES/KB Pod because it has local storage.

One option is to add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to any Pods with local storage.

I just wanted to verify, is it considered safe to do that for Elasticsearch and/or Kibana Pods, assuming that they are replicated and have a PDB defined?

[pebrc (Peter Brachwitz)]

There is no easy answer for this one. On the one hand it should be OK to remove the node, replicas will be promoted to primaries and the affected indices should become available again very quickly after the original node has been removed. The PDB should in addition protect from additional planned configuration changes that might reduce the node count temporarily even more (eg. a rolling upgrade or similar).

On the other hand you are putting the cluster at risk with this strategy if such an eviction coincides with another unplanned failure of a node. If you are unlucky and this other failure affects the replicas for the indices/shards that are gone due to the eviction you might have a data loss scenario.

The problem with local storage is that there is no good alternative unless you have some form of storage provider level copying mechanism to get the data to another node. So at the very least I would make sure to have Elasticsearch snapshots set up and configured and run at reasonably short intervals to reduce the amount of data loss you might incur.

See also Storage recommendations | Elastic Cloud on Kubernetes [master] | Elastic which covers some of this as well.

[jketcham (Jack Ketcham)]

Ok thanks for the info @pebrc!

I think we're going to be ok trying out the safe-to-evict annotation with our Elasticsearch Pods since we're using network-attached PersistentVolumes and we have automatic snapshots enabled if anything goes wrong.

Regarding local storage and having a copying mechanism, I'm guessing that would be something like using VolumeSnapshots right? GKE has that working with their CSI driver, docs here (and volume cloning too). Since we're using network-attached volumes that's not a real issue for us, but if we did want to switch to local volumes it'd be good to know what to look into (and I'd probably leave off the safe-to-evict annotation in that case).