Is it recommended to provide custom livenessProbe and readinessProbe for services deployed using the ECK operator?
I can see that the operator creates readiness probes by default. Is the default suitable for production environment? What about a liveness probe, should I define one? How do I do it?
For a stateful application such as Elasticsearch, it might not be prudent to let Kubernetes automatically restart unresponsive pods by defining a livenessProbe. It depends on your use case and risk tolerance. Indiscriminately restarting Elasticsearch nodes will have a negative impact on your cluster state and application performance depending on a variety of factors such as how you have configured Elasticsearch and how much data you have stored. Elasticsearch has its own clustering logic that can detect dead nodes and work around them so there's limited utility in using Kubernetes liveness probes in addition as well.
We are aware of several production deployments of ECK both internally and externally. The default readiness probe seems to work very well for those deployments. Again, it is up to you to decide what is suitable for your use case and change the default if necessary. Instructions on doing so can be found at https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-readiness.html.
1 Like