Uninstalling ECK - hangs forever

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello World!

I'm tried to follow Uninstalling ECK | Elastic Cloud on Kubernetes [1.0] | Elastic, however bellow commands hangs forever:

$ kubectl get namespaces --no-headers -o custom-columns=:metadata.name \
> | xargs -n1 kubectl delete elastic --all -n
elasticsearch.elasticsearch.k8s.elastic.co "quickstart" deleted
kibana.kibana.k8s.elastic.co "quickstart" deleted

Please advise.

2

Hi @alexus, if you were running an older version than 1.0 and removed the operator already, it is possible that there may be finalizers blocking the deletion. If that's the case you can use kubectl edit and remove the finalizers manually which should allow the deletions to go through.

If you're still having issues can you share the version you are running and the manifest of the resource that is hanging?

3

Having the same issue. Removing finalizer (kubernetes) in elastic-system namespace didn't help:

KUBE_EDITOR="nano" kubectl edit ns elastic-system

These commands help identifying the problems:

kubectl api-resources --verbs=list --namespace d -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n elastic-system

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n elastic-system

Trying to remove things one by one (secretes, configmaps etc)
Does somebody know the easy way to remove the operator?

4

I think this is a misunderstanding. You are editing the namespace here and are potentially removing Kubernetes internal finalizers. What Anya was referring to were finalizers that older versions of ECK set on the resources managed by ECK itself: Elasticsearch, Kibana and APMServer

If your deinstallation is getting stuck I would recommend checking those resources for finalizers especially if you were running one of the early alpha or beta versions of ECK and removing them manually.

To clarify this manual process should normally not be necessary. But I can think of scenarios where it might:

  • you ran an older pre-1.0.0 version of ECK that is no longer running/was deleted but you still have Elasticsearch/Kibana or APM Server resources
  • you paused reconciliation of Elasticsearch/Kibana/APM resources
  • you are running an older version of ECK but it is not reconciling resources (check the operator logs to find out why)

See also Troubleshoot ECK | Elastic Cloud on Kubernetes [2.10] | Elastic

5

Thanks @pebrc
k8s and eck operators are latest (1.0.1). Before removing finalizers I tried:
kubectl delete -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml
but that's where I got elastic-system namespace stuck in Terminating status...
I manually removed all resources from the namespace but it was still in Terminating
Force deleting --force --grace-period=0 didn't help, it also stuck...
The only solution that helped was this:
Starting kubectl proxy in one terminal:
kubectl proxy
In another terminal:

kubectl get ns elastic-system -o json |
jq '.spec.finalizers=' |
curl -X PUT http://localhost:8001/api/v1/namespaces/elastic-system/finalize -H "Content-Type: application/json" --data @-

But this issue might be related to Kubernetes itself:

6

Yes, you are right. I think I misunderstood your problem. In version 1.0.1 ECK does not use finalizers anymore.