Elasticsearch 6.6.0 docker container Error: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]

ShwetaP · June 24, 2019, 11:22pm

I tried upgrading elasticsearch from 6.6.0 to 7.1.1. Due to some issues I reverted back to 6.6.0 and now the elasticsearch container is having issues in starting up. Any help in troubleshooting this would be appreciated thanks!

docker logs from Elasticsearch are as follows

elasticsearch_1   | Caused by: java.lang.IllegalArgumentException: Unexpected field [cluster_uuid_committed]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1270) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.node.Node.<init>(Node.java:497) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	... 6 more
elasticsearch_1   | [2019-06-24T23:11:41,535][INFO ][o.e.x.m.p.NativeController] [6A_35An] Native controller process has stopped - no new native processes can be started

docker-compose.yml file

services:

  elasticsearch:
build:
  context: .
  dockerfile: ./dockerfiles/elasticsearch
  args:
    - ELK_VERSION

volumes:
  - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
  - esdatanew:/usr/share/elasticsearch/data
ports:
  - "9200:9200"
  - "9300:9300"
environment:
  ES_JAVA_OPTS: "-Xms1g -Xmx1g"
networks:
  - elk
restart: unless-stopped

elasticsearch.yml file

    ## Default Elasticsearch configuration from elasticsearch-docker.
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

sysctl vm.max_map_count gives vm.max_map_count = 262144

ShwetaP · June 25, 2019, 8:01pm

More log trace shows the following error.

elasticsearch_1 | [2019-06-25T19:52:56,761][ERROR][o.e.g.GatewayMetaState ] [6A_35An] failed to read local state, exiting...
elasticsearch_1 | org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]
elasticsearch_1 | at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:88) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:497) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:265) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | Caused by: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | ... 15 more

If I delete the /nodes/0 file will I loose all data? and will it solve the problem?

katariaashish · June 26, 2019, 6:27am

I am facing the same problem. However, I am using docker compose to setup my elasticsearch. I was earlier using 7.1.1 and then downgraded to 6.5.4, but I don't understand the downgrade dependency problem inside containers. I thought containers were not supposed to leave a trace after they are removed.

Can someone please explain and provide a solution?

Below is my docker-compose file:

version: '2.2'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
    container_name: elasticsearch
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

volumes:
  esdata1:
    driver: local

DavidTurner · June 26, 2019, 6:48am

You cannot downgrade an Elasticsearch node in place. Once you've run a node of a newer version it will have changed the data on disk in a way that prevents the older version from reading it. If you need to roll back an upgrade, you must start a new empty cluster and restore your data from snapshots. This is mentioned in the docs on upgrades:

Back up your data! You must have a snapshot of your data to roll back to an earlier version.

katariaashish · June 26, 2019, 6:57am

Hi @DavidTurner!
Thanks for your reply. I got around my problem after removing the volume created esdata1 in my compose file. Fortunately for me, I was only setting up my ELK stack for the first time so I didn't mind having to remove my mapped volume.

ShwetaP · July 9, 2019, 7:08pm

@DavidTurner I do have s3 backup setup but currently my elasticsearch instance is not running
Before removing my mapped volume - esdatanew:/usr/share/elasticsearch/data from the docker-compose.yml file I want to make sure that the previous data is backed up
how can I check that?

DavidTurner · July 10, 2019, 8:08am

The truly reliable way to test that a backup works is to restore it and verify that the restored data is as you expect. That's just a general good practice, nothing specifically to do with Elasticsearch.

system · August 7, 2019, 8:08am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch 6.8 in Docker crashing with SIGILL after restart - corrupt state? Elasticsearch	2	1104	October 30, 2019
Fix:upgrade elasticsearch from 6.2.2 to 7.11 in mac docker compose ,and "unsupported version: [6.4.3] minimal compatible version is: [6.8.0]" Elasticsearch	1	1254	July 23, 2019
Unable to start elasticsearch Elasticsearch	3	323	March 16, 2021
Elasticsearch reporting unhealty status Elasticsearch docker	3	4093	May 13, 2019
X-Pack / Elastic - Cannot read security-version string in index .security Elasticsearch elastic-stack-security , docker	3	989	January 1, 2021

Elasticsearch 6.6.0 docker container Error: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]

Related topics