K8S Install Fails on version < 7.2

engineertdog · August 8, 2019, 5:02pm

I am able to install v7.2 with the statefulset config that is listed below, but for a service I'm running, it hasn't been updated to support v7 yet. Currently, the service only supports v6.3. The following is the error I get when trying to boot up the statefulset.

Statefulset config

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - name: http
          containerPort: 9200
          protocol: TCP
        - name: transport
          containerPort: 9300
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: gp2
      resources:
        requests:
          storage: 100Gi

engineertdog · August 8, 2019, 5:03pm

Error log. Too long to include it in OP.

[es-cluster-0] initializing ...
[es-cluster-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]]; nested: IOException[failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]]; nested: IllegalArgumentException[[node_meta_data] unknown field [node_version], parser not found];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:331) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
... 6 more
Caused by: java.io.IOException: failed to read [id:4, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/node-4.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:325) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
... 6 more
Caused by: java.lang.IllegalArgumentException: [node_meta_data] unknown field [node_version], parser not found
at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:347) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:199) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:320) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:245) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]

DavidTurner · August 8, 2019, 7:58pm

You are attempting to downgrade a node in place. The data on disk was written by a 7.2.0 node so cannot be read by a node of an earlier version.

If this is a new installation then your best bet is to wipe the data path and start again. If you need to preserve the data then you will need to do a reindex-from-remote to get the data back into the old cluster.

engineertdog · August 9, 2019, 5:29pm

Ah of course; I don't know why I didn't think of that. When upgrading from 6.3 to 7.2+, will the new installation be able to utilize the database, or will I need to re-index? It's not mentioned in the upgrade docs.

DavidTurner · August 10, 2019, 7:17am

Upgrading in place is fine, indeed it's normally the best thing to do. It's just downgrading that isn't supported.

system · September 7, 2019, 7:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.