Elasticsearch 6.6.0 docker container Error: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]

ShwetaP · June 24, 2019, 11:22pm

I tried upgrading elasticsearch from 6.6.0 to 7.1.1. Due to some issues I reverted back to 6.6.0 and now the elasticsearch container is having issues in starting up. Any help in troubleshooting this would be appreciated thanks!

docker logs from Elasticsearch are as follows

elasticsearch_1   | Caused by: java.lang.IllegalArgumentException: Unexpected field [cluster_uuid_committed]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1270) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.node.Node.<init>(Node.java:497) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1   | 	... 6 more
elasticsearch_1   | [2019-06-24T23:11:41,535][INFO ][o.e.x.m.p.NativeController] [6A_35An] Native controller process has stopped - no new native processes can be started

docker-compose.yml file

services:

  elasticsearch:
build:
  context: .
  dockerfile: ./dockerfiles/elasticsearch
  args:
    - ELK_VERSION

volumes:
  - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
  - esdatanew:/usr/share/elasticsearch/data
ports:
  - "9200:9200"
  - "9300:9300"
environment:
  ES_JAVA_OPTS: "-Xms1g -Xmx1g"
networks:
  - elk
restart: unless-stopped

elasticsearch.yml file

    ## Default Elasticsearch configuration from elasticsearch-docker.
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

sysctl vm.max_map_count gives vm.max_map_count = 262144

ShwetaP · June 25, 2019, 8:01pm

More log trace shows the following error.

elasticsearch_1 | [2019-06-25T19:52:56,761][ERROR][o.e.g.GatewayMetaState ] [6A_35An] failed to read local state, exiting...
elasticsearch_1 | org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]
elasticsearch_1 | at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:88) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:497) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:265) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | Caused by: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | ... 15 more

If I delete the /nodes/0 file will I loose all data? and will it solve the problem?

katariaashish · June 26, 2019, 6:27am

I am facing the same problem. However, I am using docker compose to setup my elasticsearch. I was earlier using 7.1.1 and then downgraded to 6.5.4, but I don't understand the downgrade dependency problem inside containers. I thought containers were not supposed to leave a trace after they are removed.

Can someone please explain and provide a solution?

Below is my docker-compose file:

version: '2.2'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
    container_name: elasticsearch
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

volumes:
  esdata1:
    driver: local

DavidTurner · June 26, 2019, 6:48am

You cannot downgrade an Elasticsearch node in place. Once you've run a node of a newer version it will have changed the data on disk in a way that prevents the older version from reading it. If you need to roll back an upgrade, you must start a new empty cluster and restore your data from snapshots. This is mentioned in the docs on upgrades:

Back up your data! You must have a snapshot of your data to roll back to an earlier version.

katariaashish · June 26, 2019, 6:57am

Hi @DavidTurner!
Thanks for your reply. I got around my problem after removing the volume created esdata1 in my compose file. Fortunately for me, I was only setting up my ELK stack for the first time so I didn't mind having to remove my mapped volume.

ShwetaP · July 9, 2019, 7:08pm

@DavidTurner I do have s3 backup setup but currently my elasticsearch instance is not running
Before removing my mapped volume - esdatanew:/usr/share/elasticsearch/data from the docker-compose.yml file I want to make sure that the previous data is backed up
how can I check that?

DavidTurner · July 10, 2019, 8:08am

The truly reliable way to test that a backup works is to restore it and verify that the restored data is as you expect. That's just a general good practice, nothing specifically to do with Elasticsearch.

system · August 7, 2019, 8:08am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch reporting unhealty status Elasticsearch docker	3	4063	May 13, 2019
Issue while upgrading elasticsearch from 7.17.3 to 8.5.1 Elasticsearch docker	13	74	October 22, 2024
Elasticsearch statefulset deployment failed with 8.11.1 image version Elasticsearch elastic-stack-alerting , docker	1	331	December 28, 2023
Upgrade with docker-compose fails to start Elasticsearch	2	1705	August 14, 2018
Elastic search docker image upgrade not working from elasticsearch:7.1.1 to elasticsearch:7.16.1 Elasticsearch docker	2	599	January 17, 2022

Elasticsearch 6.6.0 docker container Error: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/_state/global-143.st]

Related Topics