Elastic search fails during restart with unknown setting [metadata.labels.app]

Elastic Stack Elasticsearch
1 
.plugins.PluginsService","elasticsearch.node.name":"es-cluster-1","elasticsearch.cluster.name":"k8s-logs"}
{"@timestamp":"2023-03-29T06:06:49.266Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"es-cluster-1","elasticsearch.cluster.name":"k8s-logs","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [metadata.labels.app] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [metadata.labels.app] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:559)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:505)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:475)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:445)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:463)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:316)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [apiVersion] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [data.elasticsearch.yml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [kind] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [metadata.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n"}
2

Please share more of the log and your config.

There is not enough here for anyone to be able to provide reasonable assistance on.

3

okay, i deployed ELK configuration incorrectly. looks okay now.
One more issue that i am facing now.

ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/k8s-logs.log
{"@timestamp":"2023-03-29T09:54:24.528Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"es-cluster-1","elasticsearch.cluster.name":"k8s-logs","error.type":"java.lang.IllegalStateException","error.message":"failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?","error.stack_trace":"java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:285)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:469)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:316)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /usr/share/elasticsearch/data/node.lock\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:117)\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:223)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:198)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:277)\n\t... 5 more\n"}

i doubt file share is used by other instance. how to modify deployment file for EFS mount for multiple node. can you help

4 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
spec:
  serviceName: elasticsearch
  replicas: 2
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.5.0
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: config
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          readOnly: true
          subPath: elasticsearch.yml
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      volumes:
      - name: config
        configMap:
          name: elasticsearch-config  
      - name: data
        persistentVolumeClaim:
          claimName: efs-claim-elk
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true

Deployment file used for Elasticsearch

6 
{"@timestamp":"2023-03-29T10:18:46.623Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"es-cluster-1","elasticsearch.cluster.name":"k8s-logs","error.type":"java.io.UncheckedIOException","error.message":"Failed to create persistent cache writers","error.stack_trace":"java.io.UncheckedIOException: Failed to create persistent cache writers\n\tat org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:347)\n\tat org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100)\n\tat org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:341)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.lambda$new$15(Node.java:704)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:719)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:316)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /usr/share/elasticsearch/data/snapshot_cache/write.lock\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:117)\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.apache.lucene.core@9.4.1/org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:948)\n\tat org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createCacheIndexWriter(PersistentCache.java:378)\n\tat org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:343)\n\t... 19 more\n"}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.