# Elasticsearch error while i restart the server **URL:** \ **Category:** Elasticsearch\ **Created:** [September 9, 2021, 4:19pm UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802 "2021-09-09T16:19:15Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![umangpatel](https://avatars.discourse-cdn.com/v4/letter/u/3bc359/32.png) [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Post date:** [September 9, 2021, 4:19pm UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/1 "2021-09-09T16:19:15Z")
Error=failed to build store metadata. checking segment info integrity (with commit [n I have removed the older data directly(index) from the node data directory, while i have restart the elasticsearch server and see in the log i'm getting this above error. so what is the solution of above error. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 9, 2021, 11:33pm UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/2 "2021-09-09T23:33:48Z")
Welcome to our community! 😃 Please provide more of the error/log that you are seeing. > [@umangpatel](#): > > I have removed the older data directly(index) from the node data directory What do you mean here, you delete the data directory from the disk? ---
**Author:** ![umangpatel](https://avatars.discourse-cdn.com/v4/letter/u/3bc359/32.png) [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Post date:** [September 13, 2021, 5:23am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/3 "2021-09-13T05:23:04Z")
yes remove old index in elasticsearch. whenever i try to start elasticsearch service using sudo systemctl start elasticsearch.service is not start. why ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 13, 2021, 5:25am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/4 "2021-09-13T05:25:34Z")
You should never delete data from the disk, you should always use the APIs - [Delete index API | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html) What does the full Elasticsearch log show from startup to error? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 13, 2021, 6:31am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/6 "2021-09-13T06:31:17Z")
That doesn't show any errors. Also please format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 ---
**Author:** ![umangpatel](https://avatars.discourse-cdn.com/v4/letter/u/3bc359/32.png) [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Post date:** [September 15, 2021, 10:53am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/7 "2021-09-15T10:53:02Z")
I have a ES cluster showing the status as red. { "cluster\_name" : "XYZ", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 72, "active\_shards" : 72, "relocating\_shards" : 0, "initializing\_shards" : 2, "unassigned\_shards" : 78, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0 } I have restart the server because of low disk watermark error, and delete some index through a winscp tool, a index is older like 2019 & 2020. after the restart the server i got above error and metrics is not stored in es. also i'm using apiman and in apiman error log i see adding metrics to es error. most of shard is UNASSIGNED state pls provide the solution. anyone have a solution of above error? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 15, 2021, 9:18pm UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/8 "2021-09-15T21:18:33Z")
You really need to show these errors please. ---
**Author:** ![umangpatel](https://avatars.discourse-cdn.com/v4/letter/u/3bc359/32.png) [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Post date:** [September 16, 2021, 4:41am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/9 "2021-09-16T04:41:45Z")
Below is the APIMAN logs ```auto Error adding metric to ES: ERROR [stderr] (ESMetricsConsumer) com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22 path $ ERROR [stderr] (ESMetricsConsumer) at com.google.gson.JsonParser.parse(JsonParser.java:65) ERROR [stderr] (ESMetricsConsumer) at com.google.gson.JsonParser.parse(JsonParser.java:45) ERROR [stderr] (ESMetricsConsumer) at io.searchbox.action.AbstractAction.parseResponseBody(AbstractAction.java:96) ERROR [stderr] (ESMetricsConsumer) at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:130) ERROR [stderr] (ESMetricsConsumer) at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:125) ERROR [stderr] (ESMetricsConsumer) at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:30) ERROR [stderr] (ESMetricsConsumer) at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:146) ERROR [stderr] (ESMetricsConsumer) at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:65) ERROR [stderr] (ESMetricsConsumer) at io.apiman.gateway.engine.es.ESMetrics.processQueue(ESMetrics.java:126) ERROR [stderr] (ESMetricsConsumer) at io.apiman.gateway.engine.es.ESMetrics$1.run(ESMetrics.java:100) ERROR [stderr] (ESMetricsConsumer) at java.lang.Thread.run(Thread.java:745) ERROR [stderr] (ESMetricsConsumer) Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22 path $ ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1573) ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1423) ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:546) ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.peek(JsonReader.java:429) ERROR [stderr] (ESMetricsConsumer) at com.google.gson.JsonParser.parse(JsonParser.java:60) ERROR [stderr] (ESMetricsConsumer) ... 10 more ``` below is the elasticsearch server logs ```auto [WARN][indices.cluster] [abc Two] [[test][1]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][1]: Recovery failed from [alex doe][XB0zayb3R5CCIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8QdW3ed-ziSqhpw][XYZ.test.in][inet[xx.x.xxx.xx:9300]] at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280) at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70) at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:567) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.transport.RemoteTransportException: [alex doe][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [test][1] Phase[2] Execution failed at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:902) at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:780) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125) at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.transport.RemoteTransportException: [abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog] Caused by: org.elasticsearch.index.translog.TranslogException: [test][1] failed to create new translog file at org.elasticsearch.index.translog.fs.FsTranslog.newTranslog(FsTranslog.java:253) at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:189) at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:146) at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32) at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1355) at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1350) at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:870) at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:307) at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:290) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/1/translog/translog-15784574114474 (Permission denied) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.(RandomAccessFile.java:243) at org.elasticsearch.index.translog.fs.RafReference.(RafReference.java:49) at org.elasticsearch.index.translog.fs.FsTranslog.newTranslog(FsTranslog.java:251) ... 13 more ``` Below is the clone of elasticseach server log ```auto [WARN][cluster.action.shard] [alex doe] [test][4] received shard failed for [test][4], node[gywAkbl8QdW3ed-ziSqhpw], [R], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-16], details[shard failure [failed recovery][RecoveryFailedException[[test][4]: Recovery failed from [alex doe][XB0zayb3RCCIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8QdW3ed-ziSqhpw][XYZ.test.in][inet[/xx.x.xxx.xx:9300]]]; nested: RemoteTransportException[[Baron Macabre][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test][4] Phase[2] Execution failed]; nested: RemoteTransportException[[abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]]; nested: TranslogException[[test][4] failed to create new translog file]; nested: FileNotFoundException[/test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/4/translog/translog-1574574114492 (Permission denied)]; ]]], indexUUID [rAHH1oQQygkCB2bdzqZQ], reason [shard failure [failed recovery][RecoveryFailedException[[test][4]: Recovery failed from [alex doe][XB0zayb3RC5CIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8Qd7W3ed-ziSqhpw][XYZ.test.in][inet[/xx.x.xxx.xx:9300]]]; nested: RemoteTransportException[[alex doe][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test][4] Phase[2] Execution failed]; nested: RemoteTransportException[[abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]]; nested: TranslogException[[test][4] failed to create new translog file]; nested: FileNotFoundException[/test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/4/translog/translog-15746574114492 (Permission denied)]; ]] ``` see above logs. does have any solution of above error? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 16, 2021, 5:27am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/10 "2021-09-16T05:27:45Z")
> [@umangpatel](#): > > ```auto > Caused by: java.io.FileNotFoundException: /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/1/translog/translog-15784574114474 (Permission denied) > > ``` Why are you running Elasticsearch 1.7? That's extremely well past [EOL](https://www.elastic.co/support/eol). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2021, 5:48am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/11 "2021-09-16T05:48:00Z")
Never ever delete data directly from Elasticsearch data path as this can cause severe problems you can not rwcover from - always use the APIs. ---
**Author:** ![umangpatel](https://avatars.discourse-cdn.com/v4/letter/u/3bc359/32.png) [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Post date:** [September 16, 2021, 7:40am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/12 "2021-09-16T07:40:24Z")
So now what can i do, so my cluster health is changed red to green? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 16, 2021, 8:03am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/13 "2021-09-16T08:03:29Z")
I'd run: ``` curl -XDELETE http://localhost:9200/test ``` This should remove the faulty index. If this does not work, you can stop the cluster, then run: ```auto rm -rf /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test ``` And then restart the cluster. This should work as far as I remember the 1.x version 😉 ---
**Author:** ![umangpatel](https://avatars.discourse-cdn.com/v4/letter/u/3bc359/32.png) [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Post date:** [September 20, 2021, 10:56am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/14 "2021-09-20T10:56:17Z")
```auto [2021-09-20 16:21:45,521][DEBUG][action.bulk] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2021-09-20 16:21:45,521][DEBUG][action.bulk] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2021-09-20 16:21:45,521][DEBUG][action.bulk] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2021-09-20 16:21:45,521][DEBUG][action.bulk] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] ``` what is the solution of above error? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 11:19am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/15 "2021-09-20T11:19:24Z")
It does not look like an error. I can see it as `DEBUG` level. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 18, 2021, 11:20am UTC](https://discuss.elastic.co/t/elasticsearch-error-while-i-restart-the-server/283802/16 "2021-10-18T11:20:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.