# Failed to load metata when restarting elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [January 7, 2022, 7:21am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709 "2022-01-07T07:21:30Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![bruce2](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@bruce2](https://discuss.elastic.co/u/bruce2)\ **Post date:** [January 7, 2022, 7:21am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/1 "2022-01-07T07:21:30Z")
When I restart a node of the elasticsearch cluster, it failed to start. I set log level to trace. The logs is like below: > [2022-01-05T19:2...][TRACE][o.e.g.PersistedClusterStateService] [HOSTNAME] found index metadata for [index1\_name] > [2022-01-05T19:2...][TRACE][o.e.g.PersistedClusterStateService] [HOSTNAME] processing doc 440 > [2022-01-05T19:2...][TRACE][o.e.b.Bootstrap] [HOSTNAME] Exception > org.elasticsearch.ElasticsearchExecption: failed t load metadata > at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetadataSate.java:187) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.node.Node.start(Node.java:786) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.cli.EnvironmentAwareCommand.Execute(EnvironmentAwareCommand.java:86) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127).Eeecute(EnvironmentAwareCommand.java:86) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.cli.Command.main(Command.java:90).Eeecute(EnvironmentAwareCommand.java:86) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~ [elasticsearch-7.9.2.jar:7.9.2] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~ [elasticsearch-7.9.2.jar:7.9.2] > Caused by: com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x7b) -- rather, it starts with '{' (plain JSON input?) -- can not parse > at [Source: (byte)"error": {"root\_cause": [{"type": "sql\_illegal\_argument\_exception", "reason": "The default limit [65535] for aggerate sorting has been reached; please specify a LIMIT"}], "type": }.... debugA2s?index.indexing.slowlog.threshold.index.infoA5s?index.indexing.slowlog.threshold.index.traceD500ms?index.......index.number\_of\_replicas"[truncated 1339 bytes]; line: -1, column: 0] > at com.fasterxml.jackson.dataformat.smile.SmileParseBootstrapper.constructParse(SmileParserBootstrapper.java:133) ~[jackson-dataformat-smile-2.10.4.jar:2.10.4] > ... > ... > ... At last, I backed up the elasticsearch data path, and mkdir a new empty data path, then elasticsearch is started. Does anyone know the reason of the restart failure, and how to avoid it? Thanks. It seems the error occurs when parse plain json data into smile format. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 7, 2022, 12:03pm UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/2 "2022-01-07T12:03:52Z")
> [@bruce2](#): > > Does anyone know the reason of the restart failure, and how to avoid it? Thanks. Looks like the bug fixed by [Fix DefaultRestChannel Corrupting Shared Buffers on Serialialization Issues by original-brownbear · Pull Request #72274 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/72274) - you should upgrade to pick up the fix. ---
**Author:** ![bruce2](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@bruce2](https://discuss.elastic.co/u/bruce2)\ **Post date:** [January 10, 2022, 3:17am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/3 "2022-01-10T03:17:44Z")
Thanks for your kind reply. David, I don't understand about the bug PR #72274 fixed, Would you explain what impact the bug has, and why you think my restart failure is caused by the bug? Do you know how to reproduce the restart failure? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 10, 2022, 7:51am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/4 "2022-01-10T07:51:21Z")
Writing the cluster state uses a shared buffer which in your case appears to have been corrupted by a REST response. The bug I linked explains exactly how that can happen in your version. It's a very rare race condition but yes it is possible to reproduce it. ---
**Author:** ![bruce2](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@bruce2](https://discuss.elastic.co/u/bruce2)\ **Post date:** [January 14, 2022, 6:43am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/5 "2022-01-14T06:43:50Z")
😀 Thank you, Sorry David. I don't understand all of your conversatioin in the PR page. Is there a full steps by which I can reproduce the bug? I want reproduce the issue. ---
**Author:** ![bruce2](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@bruce2](https://discuss.elastic.co/u/bruce2)\ **Post date:** [January 14, 2022, 7:02am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/6 "2022-01-14T07:02:45Z")
Now I have found the corrupted file `$(path.data)/nodes/0/_state/_10up.fdt` with the command below. ```auto grep 'sql_illegal_argument_exception' -r $(path.data)/nodes/0 ``` In my view, It's this file that stop the elasticsearch node to be start. After read the PR #72274 conversation, I got some of why the issue happened, although not clear about the detail. It seems because of the bug #72274 fixed, the response json "{"root\_cause": [{"type": "sql\_illegal\_argument\_exception.." which should only be sent as a query response was writed to the file `$(path.data)/nodes/0/_state/_10up.fdt` by mistake. Then it failed to started when we restart elasticsearch node, because the elasticsearch can't not parse the file `$(path.data)/nodes/0/_state/_10up.fdt` correctly as the error message says below. > Caused by: com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x7b) -- rather, it starts with '{' (plain JSON input?) -- can not parse > at [Source: (byte)"error": {"root\_cause": [{"type": "sql\_illegal\_argument\_exception", "reason": "The default limit [65535] for aggerate sorting has been reached; please specify a LIMIT"}], "type": }.... debugA2s?index.indexing.slowlog.threshold.index.infoA5s?index.indexing.slowlog.threshold.index.traceD500ms?index.......index.number\_of\_replicas"[truncated 1339 bytes]; line: -1, column: 0] ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 14, 2022, 7:46am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/7 "2022-01-14T07:46:17Z")
It's not easy to reproduce race conditions like the one you hit here, and I don't have the time to help you do so unfortunately. Maybe someone else will step in to help you but all I can do is recommend again that you upgrade to pick up the fix. ---
**Author:** ![bruce2](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@bruce2](https://discuss.elastic.co/u/bruce2)\ **Post date:** [January 14, 2022, 8:23am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/8 "2022-01-14T08:23:05Z")
Thank you again. I will upgrade the elasticseatch cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 11, 2022, 8:23am UTC](https://discuss.elastic.co/t/failed-to-load-metata-when-restarting-elasticsearch/293709/9 "2022-02-11T08:23:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.