How to update Elasticsearch ECS in K8S without data loss

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I use ECS (Elastic Cloud on Kubernetes) with Azure Kubernetes service. ECS version 1.3.1
I need to update Elasticsearch version from 7.10.0 to 7.10.1
I have changed the version number in yaml file and run

# kubectl apply -f elasticsearch.yaml elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch configured
But the version in not upgraded. Pod was not even restarted.

My yaml:

    apiVersion: elasticsearch.k8s.elastic.co/v1
    kind: Elasticsearch
    metadata:
      name: elasticsearch
      namespace: healthcheck
    spec:
      version: 7.10.1
      nodeSets:
      - name: elasticsearch
        count: 1
        volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
            storageClassName: default
        config:
          node.master: true
          node.data: true
          node.ingest: true
          node.store.allow_mmap: false
1 Like
2

Have you checked that the operator is actually running? Are there any errors in the operator logs?

We have a description how to get to the operator logs here https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-troubleshooting-methods.html

3

Operator is running.
I have only one pod with Elasticsearch. Perhaps this is an issue.
Operator: "Cannot restart some nodes for upgrade at this time"
Here are the logs:

    {
    	"log.level": "info",
    	"@timestamp": "2021-01-21T13:34:35.868Z",
    	"log.logger": "elasticsearch-controller",
    	"message": "Starting reconciliation run",
    	"service.version": "1.3.1+a0a0a212",
    	"service.type": "eck",
    	"ecs.version": "1.4.0",
    	"iteration": 36583,
    	"namespace": "healthcheck",
    	"es_name": "elasticsearch"
    }
    {
    	"log.level": "info",
    	"@timestamp": "2021-01-21T13:34:36.183Z",
    	"log.logger": "zen2",
    	"message": "Ensuring no voting exclusions are set",
    	"service.version": "1.3.1+a0a0a212",
    	"service.type": "eck",
    	"ecs.version": "1.4.0",
    	"namespace": "healthcheck",
    	"es_name": "elasticsearch"
    }
    {
    	"log.level": "info",
    	"@timestamp": "2021-01-21T13:34:36.208Z",
    	"log.logger": "migrate-data",
    	"message": "Setting routing allocation excludes",
    	"service.version": "1.3.1+a0a0a212",
    	"service.type": "eck",
    	"ecs.version": "1.4.0",
    	"namespace": "healthcheck",
    	"es_name": "elasticsearch",
    	"value": "none_excluded"
    }
    {
    	"log.level": "info",
    	"@timestamp": "2021-01-21T13:34:36.236Z",
    	"log.logger": "driver",
    	"message": "Cannot restart some nodes for upgrade at this time",
    	"service.version": "1.3.1+a0a0a212",
    	"service.type": "eck",
    	"ecs.version": "1.4.0",
    	"namespace": "healthcheck",
    	"es_name": "elasticsearch",
    	"failed_predicates": {
    		"require_started_replica": [
    			"elasticsearch-es-elasticsearch-0"
    		]
    	}
    }
    {
    	"log.level": "info",
    	"@timestamp": "2021-01-21T13:34:36.237Z",
    	"log.logger": "elasticsearch-controller",
    	"message": "Ending reconciliation run",
    	"service.version": "1.3.1+a0a0a212",
    	"service.type": "eck",
    	"ecs.version": "1.4.0",
    	"iteration": 36583,
    	"namespace": "healthcheck",
    	"es_name": "elasticsearch",
    	"took": 0.369245019
    }

While Elasticsearch emits this warnings a lot (have no ideas what does this error mean):

{
	"type": "server",
	"timestamp": "2021-01-21T13:05:10,771Z",
	"level": "WARN",
	"component": "o.e.x.s.a.AuthenticationService",
	"cluster.name": "elasticsearch",
	"node.name": "elasticsearch-es-elasticsearch-0",
	"message": "Authentication to realm file1 failed - Password authentication failed for elastic",
	"cluster.uuid": "qb2uCkZIRn-NVun_pZaNew",
	"node.id": "4RK8Vq8TQICvfxaz8WCAMA"
}
4

Problem is solved.
I have added one extra VM to the k8s cluster and operator upgraded the Elasticsearch.
It looks like there where not enough resources in the cluster to run the update.
I have added one extra Elasticsearch pod as well. Perhaps upgrade is just not working with a single Elasticsearch pod.