Elasticsearch 6.6.0 docker container Error: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]

I tried upgrading elasticsearch from 6.6.0 to 7.1.1. Due to some issues I reverted back to 6.6.0 and now the elasticsearch container is having issues in starting up. Any help in troubleshooting this would be appreciated thanks!

docker logs from Elasticsearch are as follows

elasticsearch_1   | Caused by: java.lang.IllegalArgumentException: Unexpected field [cluster_uuid_committed]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1270) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.node.Node.<init>(Node.java:497) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	... 6 more
elasticsearch_1   | [2019-06-24T23:11:41,535][INFO ][o.e.x.m.p.NativeController] [6A_35An] Native controller process has stopped - no new native processes can be started

docker-compose.yml file

services:

  elasticsearch:
build:
  context: .
  dockerfile: ./dockerfiles/elasticsearch
  args:
    - ELK_VERSION

volumes:
  - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
  - esdatanew:/usr/share/elasticsearch/data
ports:
  - "9200:9200"
  - "9300:9300"
environment:
  ES_JAVA_OPTS: "-Xms1g -Xmx1g"
networks:
  - elk
restart: unless-stopped

elasticsearch.yml file

    ## Default Elasticsearch configuration from elasticsearch-docker.
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

sysctl vm.max_map_count gives vm.max_map_count = 262144

More log trace shows the following error.

elasticsearch_1 | [2019-06-25T19:52:56,761][ERROR][o.e.g.GatewayMetaState ] [6A_35An] failed to read local state, exiting...
elasticsearch_1 | org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]
elasticsearch_1 | at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:88) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:497) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:265) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | Caused by: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | ... 15 more

If I delete the /nodes/0 file will I loose all data? and will it solve the problem?

I am facing the same problem. However, I am using docker compose to setup my elasticsearch. I was earlier using 7.1.1 and then downgraded to 6.5.4, but I don't understand the downgrade dependency problem inside containers. I thought containers were not supposed to leave a trace after they are removed.

Can someone please explain and provide a solution?

Below is my docker-compose file:

version: '2.2'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
    container_name: elasticsearch
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

volumes:
  esdata1:
    driver: local

You cannot downgrade an Elasticsearch node in place. Once you've run a node of a newer version it will have changed the data on disk in a way that prevents the older version from reading it. If you need to roll back an upgrade, you must start a new empty cluster and restore your data from snapshots. This is mentioned in the docs on upgrades:

  1. Back up your data! You must have a snapshot of your data to roll back to an earlier version.
1 Like

Hi @DavidTurner!
Thanks for your reply. I got around my problem after removing the volume created esdata1 in my compose file. Fortunately for me, I was only setting up my ELK stack for the first time so I didn't mind having to remove my mapped volume.

@DavidTurner I do have s3 backup setup but currently my elasticsearch instance is not running
Before removing my mapped volume - esdatanew:/usr/share/elasticsearch/data from the docker-compose.yml file I want to make sure that the previous data is backed up
how can I check that?

The truly reliable way to test that a backup works is to restore it and verify that the restored data is as you expect. That's just a general good practice, nothing specifically to do with Elasticsearch.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.