K8S Install Fails on version < 7.2

engineertdog · August 8, 2019, 5:02pm

I am able to install v7.2 with the statefulset config that is listed below, but for a service I'm running, it hasn't been updated to support v7 yet. Currently, the service only supports v6.3. The following is the error I get when trying to boot up the statefulset.

Statefulset config

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - name: http
          containerPort: 9200
          protocol: TCP
        - name: transport
          containerPort: 9300
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: gp2
      resources:
        requests:
          storage: 100Gi

engineertdog · August 8, 2019, 5:03pm

Error log. Too long to include it in OP.

[es-cluster-0] initializing ...
[es-cluster-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]]; nested: IOException[failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]]; nested: IllegalArgumentException[[node_meta_data] unknown field [node_version], parser not found];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:331) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
... 6 more
Caused by: java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:325) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
... 6 more
Caused by: java.lang.IllegalArgumentException: [node_meta_data] unknown field [node_version], parser not found
at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:347) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:199) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:320) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]

DavidTurner · August 8, 2019, 7:58pm

You are attempting to downgrade a node in place. The data on disk was written by a 7.2.0 node so cannot be read by a node of an earlier version.

If this is a new installation then your best bet is to wipe the data path and start again. If you need to preserve the data then you will need to do a reindex-from-remote to get the data back into the old cluster.

engineertdog · August 9, 2019, 5:29pm

Ah of course; I don't know why I didn't think of that. When upgrading from 6.3 to 7.2+, will the new installation be able to utilize the database, or will I need to re-index? It's not mentioned in the upgrade docs.

DavidTurner · August 10, 2019, 7:17am

Upgrading in place is fine, indeed it's normally the best thing to do. It's just downgrading that isn't supported.

system · September 7, 2019, 7:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Trying to deploy statefulSet of Elasticsearch, persisting config and keystore not work Elasticsearch docker	0	120	May 31, 2024
Upgrade to 8.0.0 failed Elasticsearch docker	3	1270	April 5, 2022
Elasticsearch statefulset deployment failed with 8.11.1 image version Elasticsearch elastic-stack-alerting , docker	1	331	December 28, 2023
Elasticsearch pod stuck in CrashLoopBackOff Elastic Cloud on Kubernetes (ECK)	5	2524	November 4, 2022
After creating Elasticsearch, and then changing the yaml file, apply fails Elastic Cloud on Kubernetes (ECK)	3	1115	March 25, 2021

K8S Install Fails on version < 7.2

Related topics