# ES Data node global metadata not found
**URL:** https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452
**Category:** Elasticsearch
**Tags:** docker
**Created:** [February 5, 2021, 6:48pm UTC](https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452 "2021-02-05T18:48:18Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Gratien](https://discuss.elastic.co/u/Gratien)
#### Post date: [February 5, 2021, 6:48pm UTC](https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452/1 "2021-02-05T18:48:18Z")
Hello everyone,
I'm working on a marathon hosted cluster (7.6.2) which is not very stable sometimes.
Occasionnaly when there is a host failure, I end up with the following on a data node :
```auto
java.lang.IllegalStateException: no global metadata found in [/usr/share/elasticsearch/data/nodes/0]
```
I found very little info, and especially I have found NO mention of such a problem in here.
When I google this specific message, only ES sources show up.
Where/how exactly is stored global metadata ?
Is there someting I can do besides resetting the node and losing data ?
Please let me know if you need more context or anything,
and thanks in advance for your answers !
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 5, 2021, 6:57pm UTC](https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452/2 "2021-02-05T18:57:32Z")
I've seen this a couple of times but never in a controlled situation and there's always been something suspicious about the environment (host failures etc) that impossible to determine whether the blame lies with the infrastructure or with Elasticsearch. We added better error reporting in this PR:
>
>
> It is disastrous if we commit an incremental cluster state update
> without havin…g written the full state first. We assert that this doesn't
> happen, but it is hard to fully test the myriad ways that things might
> fail in a messy production environment. Given the disastrous
> consequences it is worth erring on the side of caution in this area.
> This commit fails invalid writes even if assertions are disabled.
Since adding that check I've never seen it fail. Would you upgrade to 7.10 to incorporate this check and then report back when it happens again?
> [@Gratien](#):
>
> Is there someting I can do besides resetting the node and losing data ?
Not really, no, although you can restore from snapshots to minimise any loss of data.
---
### Author:  [@Gratien](https://discuss.elastic.co/u/Gratien)
#### Post date: [February 5, 2021, 7:38pm UTC](https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452/3 "2021-02-05T19:38:20Z")
> [@DavidTurner](#):
>
> Would you upgrade to 7.10 to incorporate this check and then report back when it happens again?
I'll have to check for impacts and discuss it with coworkers, but I'll take it into consideration.
> [@DavidTurner](#):
>
> Not really, no, although you can restore from snapshots to minimise any loss of data.
That I will definitely do.
Would you still describe how global metadata is stored ?
In any case, thanks for your fast answer and the info, from me and the future googler 🙂
And if I go along with the upgrade and something shows up, I'll definitely post it here.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 5, 2021, 7:42pm UTC](https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452/4 "2021-02-05T19:42:04Z")
> [@Gratien](#):
>
> Would you still describe how global metadata is stored ?
All the cluster metadata is stored in a Lucene index, with one document for the global metadata and one document for the metadata for each index. This comment gives a good summary:
>
The exception `no global metadata found` means that the document for the global metadata simply isn't there, which ought to be impossible (without external tampering at least).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 5, 2021, 7:42pm UTC](https://discuss.elastic.co/t/es-data-node-global-metadata-not-found/263452/5 "2021-03-05T19:42:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.