ES Data node global metadata not found

Hello everyone,

I'm working on a marathon hosted cluster (7.6.2) which is not very stable sometimes.
Occasionnaly when there is a host failure, I end up with the following on a data node :

java.lang.IllegalStateException: no global metadata found in [/usr/share/elasticsearch/data/nodes/0]

I found very little info, and especially I have found NO mention of such a problem in here.
When I google this specific message, only ES sources show up.

Where/how exactly is stored global metadata ?
Is there someting I can do besides resetting the node and losing data ?

Please let me know if you need more context or anything,
and thanks in advance for your answers !

I've seen this a couple of times but never in a controlled situation and there's always been something suspicious about the environment (host failures etc) that impossible to determine whether the blame lies with the infrastructure or with Elasticsearch. We added better error reporting in this PR:

Since adding that check I've never seen it fail. Would you upgrade to 7.10 to incorporate this check and then report back when it happens again?

Not really, no, although you can restore from snapshots to minimise any loss of data.

I'll have to check for impacts and discuss it with coworkers, but I'll take it into consideration.

That I will definitely do.

Would you still describe how global metadata is stored ?

In any case, thanks for your fast answer and the info, from me and the future googler :slight_smile:
And if I go along with the upgrade and something shows up, I'll definitely post it here.

All the cluster metadata is stored in a Lucene index, with one document for the global metadata and one document for the metadata for each index. This comment gives a good summary:

The exception no global metadata found means that the document for the global metadata simply isn't there, which ought to be impossible (without external tampering at least).

1 Like