K8S Install Fails on version < 7.2

I am able to install v7.2 with the statefulset config that is listed below, but for a service I'm running, it hasn't been updated to support v7 yet. Currently, the service only supports v6.3. The following is the error I get when trying to boot up the statefulset.

Statefulset config

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - name: http
          containerPort: 9200
          protocol: TCP
        - name: transport
          containerPort: 9300
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: gp2
      resources:
        requests:
          storage: 100Gi

Error log. Too long to include it in OP.

[es-cluster-0] initializing ...
[es-cluster-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]]; nested: IOException[failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]]; nested: IllegalArgumentException[[node_meta_data] unknown field [node_version], parser not found];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:331) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
... 6 more
Caused by: java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:325) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
... 6 more
Caused by: java.lang.IllegalArgumentException: [node_meta_data] unknown field [node_version], parser not found
at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:347) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:199) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:320) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]

You are attempting to downgrade a node in place. The data on disk was written by a 7.2.0 node so cannot be read by a node of an earlier version.

If this is a new installation then your best bet is to wipe the data path and start again. If you need to preserve the data then you will need to do a reindex-from-remote to get the data back into the old cluster.

1 Like

Ah of course; I don't know why I didn't think of that. When upgrading from 6.3 to 7.2+, will the new installation be able to utilize the database, or will I need to re-index? It's not mentioned in the upgrade docs.

Upgrading in place is fine, indeed it's normally the best thing to do. It's just downgrading that isn't supported.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.