@DavidTurner
The IP addresses that es-4 is trying to connect were correct and no connectivity problems.
Those IPs belong to the nodes in the ES cluster.
FYI: es-{0-4} are statefulsets.
We have 5 pods es-{0-4} in the ES cluster. The elasticsearch operator is managing them with version 1.0.
The latest version is 1.5.
Ref Upgrade ECK | Elastic Cloud on Kubernetes [1.5] | Elastic
As part of the operator upgrade, we have changed the operator version and deployed it.
The operator started recreating pods. First, it kicked off es-4 and it recreated but it is not able to connect to the existing cluster. There are no network connectivity issues.
To test the changes, I tried the same in our test env. The operator recreated es-4 and I had to delete another 2 more nodes/pods[es-3, es-2] to recreate[to satisfy the quorum spec] and form a cluster.
The rest of the nodes/pods es-1 and es-0 taken care of by the operator, those were recreated and joined without any manual deletions.
Is that expected behavior? Do we need to manually kill/delete the pods to join the cluster back?