# Elasticsearch state "Invalid" after trying to install plugin **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [January 23, 2020, 4:51pm UTC](https://discuss.elastic.co/t/elasticsearch-state-invalid-after-trying-to-install-plugin/216296 "2020-01-23T16:51:59Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![radw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radw/32/53012_2.png) [@radw](https://discuss.elastic.co/u/radw)\ **Post date:** [January 23, 2020, 4:51pm UTC](https://discuss.elastic.co/t/elasticsearch-state-invalid-after-trying-to-install-plugin/216296/1 "2020-01-23T16:51:59Z")
Hi folks! I did an ECK installation on GKE with v1beta1. After 1 month I've resized the PVs to accomodate the new storage needs. Everything went smooth and the PVs resized correctly and are shown in the monitoring section of Kibana with the correct value. A few days ago I've been tasked to backup the indices to GCS so for that I needed to install the repository-gcs plugin. I've followed the guide on [elastic.co](http://elastic.co) but seems like something went wrong and now when I call kubectl get elasticsearch the output says ` PHASE Invalid` Whatever I tried to do to restore it's health it's useless. I have 3 master nodes so I tried to bring down 1 master node to see if the new pod will spin up correctly but, unfortunately, the pod is spinned up but doesn't join the cluster. My configuration is as follows: > Name: siem-test > Namespace: default > Labels: > Annotations: [common.k8s.elastic.co/controller-version:](http://common.k8s.elastic.co/controller-version:) 0.0.0-UNKNOWN > [kubectl.kubernetes.io/last-applied-configuration:](http://kubectl.kubernetes.io/last-applied-configuration:) > {"apiVersion":"[elasticsearch.k8s.elastic.co/v1beta1","kind":"Elasticsearch","metadata":{"annotations":{},"name":"siem-test","namespace](http://elasticsearch.k8s.elastic.co/v1beta1%22,%22kind%22:%22Elasticsearch%22,%22metadata%22:%7B%22annotations%22:%7B%7D,%22name%22:%22siem-test%22,%22namespace)":... > API Version: [elasticsearch.k8s.elastic.co/v1beta1](http://elasticsearch.k8s.elastic.co/v1beta1) > Kind: Elasticsearch > Metadata: > Creation Timestamp: 2019-10-25T08:44:03Z > Generation: 23 > Resource Version: 27989855 > Self Link: /apis/elasticsearch.k8s.elastic.co/v1beta1/namespaces/default/elasticsearches/siem-test > UID: 98f623e5-f703-11e9-903e-42010a8401ee > Spec: > Http: > Service: > Metadata: > Creation Timestamp: > Spec: > Ports: > Port: 9200 > Target Port: 9200 > Type: LoadBalancer > Tls: > Certificate: > Node Sets: > Config: > Node . Data: false > Node . Ingest: false > Node . Master: true > Node . Ml: false > Node . Store . Allow Mmap: false > Xpack . Security . Authc . Realms: > Native: > Native 1: > Order: 1 > Count: 3 > Name: node-master > Pod Template: > Metadata: > Labels: > Es: master-node > Spec: > Containers: > Env: > Name: ES\_JAVA\_OPTS > Value: -Xms2g -Xmx2g > Name: elasticsearch > Resources: > Limits: > Cpu: 1 > Memory: 4Gi > Init Containers: > Command: > sh > -c > bin/elasticsearch-plugin install --batch repository-gcs > > ``` > Name: install-plugins > Volume Claim Templates: > Metadata: > Name: elasticsearch-data > Spec: > Access Modes: > ReadWriteOnce > Resources: > Requests: > Storage: 10Gi > Storage Class Name: ssd > Config: > Node . Data: true > Node . Ingest: true > Node . Master: false > Node . Ml: true > Node . Store . Allow Mmap: false > Xpack . Security . Authc . Realms: > Native: > Native 1: > Order: 1 > Count: 5 > Name: node-data > Pod Template: > Metadata: > Labels: > Es: data-node > Spec: > Containers: > Env: > Name: ES_JAVA_OPTS > Value: -Xms2g -Xmx2g > Name: elasticsearch > Resources: > Limits: > Cpu: 2 > Memory: 4Gi > Init Containers: > Command: > sh > -c > bin/elasticsearch-plugin install --batch repository-gcs > > Name: install-plugins > Volume Claim Templates: > Metadata: > Name: elasticsearch-data > Spec: > Access Modes: > ReadWriteOnce > Resources: > Requests: > Storage: 200Gi > Storage Class Name: ssd > > ``` > > Secure Settings: > Secret Name: gcs-credentials > Update Strategy: > Change Budget: > Version: 7.4.0 > Status: > Available Nodes: 7 > Health: green > Phase: Invalid > Events: I don't want to lose any of the data already stored on the nodes. I have even manually installed the repository-gcs plugin on each pod but as I have to restart the elasticsearch service, it is not recognised in the Kibana interface so I can backup the indices and redo the whole cluster. Any thoughts on how can I narrow down this problem and try to solve it? Any help is highly appreciated! Radu ---
**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\ **Post date:** [January 23, 2020, 5:31pm UTC](https://discuss.elastic.co/t/elasticsearch-state-invalid-after-trying-to-install-plugin/216296/2 "2020-01-23T17:31:27Z")
There's a few places I'd recommend checking. You can do a `describe` on the ES resource and the pods to see if there are any events that may be relevant. For the new pods that are failing to join the cluster, you can look at the container logs to see why that might be happening. And you can also look at the operator logs to see why the phase may be coming up as invalid. See the troubleshooting doc here for more info on each of these steps: [https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-troubleshooting.html](https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-troubleshooting.html) It may also be worth updating to the 1.0 version of the operator as there are a variety of bug fixes. ---
**Author:** ![radw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radw/32/53012_2.png) [@radw](https://discuss.elastic.co/u/radw)\ **Post date:** [January 27, 2020, 12:30pm UTC](https://discuss.elastic.co/t/elasticsearch-state-invalid-after-trying-to-install-plugin/216296/3 "2020-01-27T12:30:42Z")
Hey @Anya_Sabo I tried going through all of the troubleshooting steps but I'm at a dead end. I've pulled this logs from the elastic-operator though: > ``` > > {"level":"info","@timestamp":"2020-01-27T08:55:47.487Z","logger":"controller-runtime.manager","message":"starting metrics server","ver":"1.0.0-beta1-84792e30","path":"/metrics"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"apmserver-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"license-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"kibana-association-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"elasticsearch-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"kibana-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"trial-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.587Z","logger":"controller-runtime.controller","message":"Starting Controller","ver":"1.0.0-beta1-84792e30","controller":"apm-es-association-controller"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"license-controller","worker count":1} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"license-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"kibana-association-controller","worker count":1} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"kibana-association-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test-exposed"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"apmserver-controller","worker count":1} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"elasticsearch-controller","worker count":1} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"annotation","message":"Resource was created with older version of operator, will not take action","ver":"1.0.0-beta1-84792e30","controller_version":"1.0.0-beta1","resource_controller_version":"0.0.0-UNKNOWN","namespace":"default","name":"siem-test-exposed"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"kibana-association-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test-exposed","took":0.000577217} > > {"level":"debug","@timestamp":"2020-01-27T08:55:47.689Z","logger":"controller-runtime.controller","message":"Successfully Reconciled","ver":"1.0.0-beta1-84792e30","controller":"kibana-association-controller","request":"default/siem-test-exposed"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.688Z","logger":"elasticsearch-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"kibana-controller","worker count":1} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"license-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test","took":0.001165524} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"annotation","message":"Resource was created with older version of operator, will not take action","ver":"1.0.0-beta1-84792e30","controller_version":"1.0.0-beta1","resource_controller_version":"0.0.0-UNKNOWN","namespace":"default","name":"siem-test"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"elasticsearch-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test","took":0.000539069} > > {"level":"debug","@timestamp":"2020-01-27T08:55:47.689Z","logger":"controller-runtime.controller","message":"Successfully Reconciled","ver":"1.0.0-beta1-84792e30","controller":"elasticsearch-controller","request":"default/siem-test"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"kibana-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test-exposed"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"annotation","message":"Resource was created with older version of operator, will not take action","ver":"1.0.0-beta1-84792e30","controller_version":"1.0.0-beta1","resource_controller_version":"0.0.0-UNKNOWN","namespace":"default","name":"siem-test-exposed"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"kibana-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-84792e30","iteration":1,"namespace":"default","name":"siem-test-exposed","took":0.00036311} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"trial-controller","worker count":1} > > {"level":"debug","@timestamp":"2020-01-27T08:55:47.689Z","logger":"controller-runtime.controller","message":"Successfully Reconciled","ver":"1.0.0-beta1-84792e30","controller":"kibana-controller","request":"default/siem-test-exposed"} > > {"level":"info","@timestamp":"2020-01-27T08:55:47.689Z","logger":"controller-runtime.controller","message":"Starting workers","ver":"1.0.0-beta1-84792e30","controller":"apm-es-association-controller","worker count":1} > > ``` I can see the operator thinks it's version is 0.0.0-UNKNOWN but I haven't tried to do any upgrade or something similar. Maybe it has something to do with how I defined the secureSettings when trying to install the repository-gcs although this error has been there since before I tried to install this plugin and inserting the gcs-credentials?! I presume the secureSettings option to have something to do with this because I've read a blog post [here](https://www.gitmemory.com/issue/elastic/cloud-on-k8s/2039/545021989) and [this github issue](https://github.com/elastic/cloud-on-k8s/issues/1666). Now I'm wondering, if I will update the ECK deployment to the latest version, could this cause a total failure of my cluster and data unrecoverable (I have PVEs but I don't know how could I mount these existing PVEs to a new ES cluster's data nodes so that the existing data will pop-up in the newly created cluster). Thank you for your support! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:41am UTC](https://discuss.elastic.co/t/elasticsearch-state-invalid-after-trying-to-install-plugin/216296/4 "2022-11-04T07:41:57Z")