ES Data node global metadata not found

Gratien · February 5, 2021, 6:48pm

Hello everyone,

I'm working on a marathon hosted cluster (7.6.2) which is not very stable sometimes.
Occasionnaly when there is a host failure, I end up with the following on a data node :

java.lang.IllegalStateException: no global metadata found in [/usr/share/elasticsearch/data/nodes/0]

I found very little info, and especially I have found NO mention of such a problem in here.
When I google this specific message, only ES sources show up.

Where/how exactly is stored global metadata ?
Is there someting I can do besides resetting the node and losing data ?

Please let me know if you need more context or anything,
and thanks in advance for your answers !

DavidTurner · February 5, 2021, 6:57pm

I've seen this a couple of times but never in a controlled situation and there's always been something suspicious about the environment (host failures etc) that impossible to determine whether the blame lies with the infrastructure or with Elasticsearch. We added better error reporting in this PR:

github.com/elastic/elasticsearch

Fail invalid incremental cluster state writes

elastic:master ← DaveCTurner:2020-08-12-fail-invalid-incremental-cluster-state-writes

opened 01:32PM - 12 Aug 20 UTC

DaveCTurner

+19 -3

It is disastrous if we commit an incremental cluster state update without havin…g written the full state first. We assert that this doesn't happen, but it is hard to fully test the myriad ways that things might fail in a messy production environment. Given the disastrous consequences it is worth erring on the side of caution in this area. This commit fails invalid writes even if assertions are disabled.

Since adding that check I've never seen it fail. Would you upgrade to 7.10 to incorporate this check and then report back when it happens again?

Not really, no, although you can restore from snapshots to minimise any loss of data.

Gratien · February 5, 2021, 7:38pm

I'll have to check for impacts and discuss it with coworkers, but I'll take it into consideration.

That I will definitely do.

Would you still describe how global metadata is stored ?

In any case, thanks for your fast answer and the info, from me and the future googler
And if I go along with the upgrade and something shows up, I'll definitely post it here.

DavidTurner · February 5, 2021, 7:42pm

All the cluster metadata is stored in a Lucene index, with one document for the global metadata and one document for the metadata for each index. This comment gives a good summary:

github.com

elastic/elasticsearch/blob/a92a647b9f17d1bddf5c707490a19482c273eda3/server/src/main/java/org/elasticsearch/gateway/PersistedClusterStateService.java#L86-L110


      
          /**
           * Stores cluster metadata in a bare Lucene index (per data path) split across a number of documents. This is used by master-eligible nodes
           * to record the last-accepted cluster state during publication. The metadata is written incrementally where possible, leaving alone any
           * documents that have not changed. The index has the following fields:
           *
           * +------------------------------+-----------------------------+----------------------------------------------+
           * | "type" (string field)        | "index_uuid" (string field) | "data" (stored binary field in SMILE format) |
           * +------------------------------+-----------------------------+----------------------------------------------+
           * | GLOBAL_TYPE_NAME == "global" | (omitted)                   | Global metadata                              |
           * | INDEX_TYPE_NAME  == "index"  | Index UUID                  | Index metadata                               |
           * +------------------------------+-----------------------------+----------------------------------------------+
           *
           * Additionally each commit has the following user data:
           *
           * +---------------------------+-------------------------+-------------------------------------------------------------------------------+
           * |        Key symbol         |       Key literal       |                                     Value                                     |
           * +---------------------------+-------------------------+-------------------------------------------------------------------------------+
           * | CURRENT_TERM_KEY          | "current_term"          | Node's "current" term (≥ last-accepted term and the terms of all sent joins)  |
           * | LAST_ACCEPTED_VERSION_KEY | "last_accepted_version" | The cluster state version corresponding with the persisted metadata           |
           * | NODE_ID_KEY               | "node_id"               | The (persistent) ID of the node that wrote this metadata                      |

This file has been truncated. show original

The exception no global metadata found means that the document for the global metadata simply isn't there, which ought to be impossible (without external tampering at least).

system · March 5, 2021, 7:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.