# Elasticsearch 6.6.0 docker container Error: failed to read \[id:143, file:/usr/share/elasticsearch/data/nodes/0/\_state/global-143.st\]

**URL:** <https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [June 24, 2019, 11:22pm UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203 "2019-06-24T23:22:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ShwetaP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwetap/32/48680_2.png) [@ShwetaP](https://discuss.elastic.co/u/ShwetaP)\
**Post date:** [June 24, 2019, 11:22pm UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/1 "2019-06-24T23:22:52Z")

</div>

I tried upgrading elasticsearch from 6.6.0 to 7.1.1. Due to some issues I reverted back to 6.6.0 and now the elasticsearch container is having issues in starting up. Any help in troubleshooting this would be appreciated thanks!

docker logs from Elasticsearch are as follows

```
elasticsearch_1 | Caused by: java.lang.IllegalArgumentException: Unexpected field [cluster_uuid_committed]
elasticsearch_1 | at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1270) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:497) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
elasticsearch_1 | ... 6 more
elasticsearch_1 | [2019-06-24T23:11:41,535][INFO][o.e.x.m.p.NativeController] [6A_35An] Native controller process has stopped - no new native processes can be started

```

docker-compose.yml file

```
services:

  elasticsearch:
build:
  context: .
  dockerfile: ./dockerfiles/elasticsearch
  args:
    - ELK_VERSION

volumes:
  - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
  - esdatanew:/usr/share/elasticsearch/data
ports:
  - "9200:9200"
  - "9300:9300"
environment:
  ES_JAVA_OPTS: "-Xms1g -Xmx1g"
networks:
  - elk
restart: unless-stopped

```

elasticsearch.yml file

```
    ## Default Elasticsearch configuration from elasticsearch-docker.
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

```

`sysctl vm.max_map_count` gives `vm.max_map_count = 262144`

---

<div class="post-metadata">

**Author:** ![ShwetaP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwetap/32/48680_2.png) [@ShwetaP](https://discuss.elastic.co/u/ShwetaP)\
**Post date:** [June 25, 2019, 8:01pm UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/2 "2019-06-25T20:01:31Z")

</div>

More log trace shows the following error.

> elasticsearch\_1 | [2019-06-25T19:52:56,761][ERROR][o.e.g.GatewayMetaState] [6A\_35An] failed to read local state, exiting...  
> elasticsearch\_1 | org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/\_state/global-143.st]  
> elasticsearch\_1 | at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:88) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.node.Node.(Node.java:497) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.node.Node.(Node.java:265) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | Caused by: java.io.IOException: failed to read [id:143, file:/usr/share/elasticsearch/data/nodes/0/\_state/global-143.st]  
> elasticsearch\_1 | at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.6.0.jar:6.6.0]  
> elasticsearch\_1 | ... 15 more

If I delete the /nodes/0 file will I loose all data? and will it solve the problem?

---

<div class="post-metadata">

**Author:** ![katariaashish](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@katariaashish](https://discuss.elastic.co/u/katariaashish)\
**Post date:** [June 26, 2019, 6:27am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/3 "2019-06-26T06:27:04Z")

</div>

I am facing the same problem. However, I am using docker compose to setup my elasticsearch. I was earlier using 7.1.1 and then downgraded to 6.5.4, but I don't understand the downgrade dependency problem inside containers. I thought containers were not supposed to leave a trace after they are removed.

Can someone please explain and provide a solution?

Below is my docker-compose file:

```
version: '2.2'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
    container_name: elasticsearch
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

volumes:
  esdata1:
    driver: local
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 26, 2019, 6:48am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/4 "2019-06-26T06:48:34Z")

</div>

You cannot downgrade an Elasticsearch node in place. Once you've run a node of a newer version it will have changed the data on disk in a way that prevents the older version from reading it. If you need to roll back an upgrade, you must start a new empty cluster and restore your data from snapshots. This is mentioned in the [docs on upgrades](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/setup-upgrade.html#_preparing_to_upgrade):

> 1. [Back up your data!](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/modules-snapshots.html) You must have a snapshot of your data to roll back to an earlier version.

---

<div class="post-metadata">

**Author:** ![katariaashish](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@katariaashish](https://discuss.elastic.co/u/katariaashish)\
**Post date:** [June 26, 2019, 6:57am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/5 "2019-06-26T06:57:51Z")

</div>

Hi @DavidTurner!  
Thanks for your reply. I got around my problem after removing the volume created `esdata1` in my compose file. Fortunately for me, I was only setting up my ELK stack for the first time so I didn't mind having to remove my mapped volume.

---

<div class="post-metadata">

**Author:** ![ShwetaP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwetap/32/48680_2.png) [@ShwetaP](https://discuss.elastic.co/u/ShwetaP)\
**Post date:** [July 9, 2019, 7:08pm UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/6 "2019-07-09T19:08:11Z")

</div>

@DavidTurner I do have s3 backup setup but currently my elasticsearch instance is not running  
Before removing my mapped volume `- esdatanew:/usr/share/elasticsearch/data` from the docker-compose.yml file I want to make sure that the previous data is backed up  
how can I check that?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 10, 2019, 8:08am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/7 "2019-07-10T08:08:13Z")

</div>

The truly reliable way to test that a backup works is to restore it and verify that the restored data is as you expect. That's just a general good practice, nothing specifically to do with Elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2019, 8:08am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-0-docker-container-error-failed-to-read-id-143-file-usr-share-elasticsearch-data-nodes-0-state-global-143-st/187203/8 "2019-08-07T08:08:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.