Elasticsearch error while i restart the server

Error=failed to build store metadata. checking segment info integrity (with commit [n

I have removed the older data directly(index) from the node data directory, while i have restart the elasticsearch server and see in the log i'm getting this above error. so what is the solution of above error.

Welcome to our community! :smiley:

Please provide more of the error/log that you are seeing.

What do you mean here, you delete the data directory from the disk?

yes remove old index in elasticsearch. whenever i try to start elasticsearch service using sudo systemctl start elasticsearch.service is not start. why ?

You should never delete data from the disk, you should always use the APIs - Delete index API | Elasticsearch Guide [7.14] | Elastic

What does the full Elasticsearch log show from startup to error?

That doesn't show any errors.

Also please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

I have a ES cluster showing the status as red.

{ "cluster_name" : "XYZ", "status" : "red", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 72, "active_shards" : 72, "relocating_shards" : 0, "initializing_shards" : 2, "unassigned_shards" : 78, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0 }

I have restart the server because of low disk watermark error, and delete some index through a winscp tool, a index is older like 2019 & 2020. after the restart the server i got above error and metrics is not stored in es. also i'm using apiman and in apiman error log i see adding metrics to es error. most of shard is UNASSIGNED state pls provide the solution. anyone have a solution of above error?

You really need to show these errors please.

Below is the APIMAN logs

Error adding metric to ES:
 ERROR [stderr] (ESMetricsConsumer) com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22 path $
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.JsonParser.parse(JsonParser.java:65)
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.JsonParser.parse(JsonParser.java:45)
 ERROR [stderr] (ESMetricsConsumer) 	at io.searchbox.action.AbstractAction.parseResponseBody(AbstractAction.java:96)
 ERROR [stderr] (ESMetricsConsumer) 	at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:130)
 ERROR [stderr] (ESMetricsConsumer) 	at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:125)
 ERROR [stderr] (ESMetricsConsumer) 	at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:30)
 ERROR [stderr] (ESMetricsConsumer) 	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:146)
 ERROR [stderr] (ESMetricsConsumer) 	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:65)
 ERROR [stderr] (ESMetricsConsumer) 	at io.apiman.gateway.engine.es.ESMetrics.processQueue(ESMetrics.java:126)
 ERROR [stderr] (ESMetricsConsumer) 	at io.apiman.gateway.engine.es.ESMetrics$1.run(ESMetrics.java:100)
 ERROR [stderr] (ESMetricsConsumer) 	at java.lang.Thread.run(Thread.java:745)
 ERROR [stderr] (ESMetricsConsumer) Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22 path $
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1573)
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1423)
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:546)
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.stream.JsonReader.peek(JsonReader.java:429)
 ERROR [stderr] (ESMetricsConsumer) 	at com.google.gson.JsonParser.parse(JsonParser.java:60) 
 ERROR [stderr] (ESMetricsConsumer) 	... 10 more

below is the elasticsearch server logs

[WARN ][indices.cluster          ] [abc Two] [[test][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [test][1]: Recovery failed from [alex doe][XB0zayb3R5CCIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8QdW3ed-ziSqhpw][XYZ.test.in][inet[xx.x.xxx.xx:9300]]
        at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280)
        at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70)
        at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:567)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: [alex doe][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [test][1] Phase[2] Execution failed
        at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:902)
        at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:780)
        at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125)
        at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49)
        at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146)
        at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: [abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]
Caused by: org.elasticsearch.index.translog.TranslogException: [test][1] failed to create new translog file
        at org.elasticsearch.index.translog.fs.FsTranslog.newTranslog(FsTranslog.java:253)
        at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:189)
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:146)
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1355)
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1350)
        at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:870)
        at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:307)
        at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:290)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/1/translog/translog-15784574114474 (Permission denied)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.elasticsearch.index.translog.fs.RafReference.<init>(RafReference.java:49)
        at org.elasticsearch.index.translog.fs.FsTranslog.newTranslog(FsTranslog.java:251)
        ... 13 more

Below is the clone of elasticseach server log

[WARN ][cluster.action.shard     ] [alex doe] [test][4] received shard failed for [test][4], node[gywAkbl8QdW3ed-ziSqhpw], [R], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-16], details[shard failure [failed recovery][RecoveryFailedException[[test][4]: Recovery failed from [alex doe][XB0zayb3RCCIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8QdW3ed-ziSqhpw][XYZ.test.in][inet[/xx.x.xxx.xx:9300]]]; nested: RemoteTransportException[[Baron Macabre][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test][4] Phase[2] Execution failed]; nested: RemoteTransportException[[abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]]; nested: TranslogException[[test][4] failed to create new translog file]; nested: FileNotFoundException[/test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/4/translog/translog-1574574114492 (Permission denied)]; ]]], indexUUID [rAHH1oQQygkCB2bdzqZQ], reason [shard failure [failed recovery][RecoveryFailedException[[test][4]: Recovery failed from [alex doe][XB0zayb3RC5CIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8Qd7W3ed-ziSqhpw][XYZ.test.in][inet[/xx.x.xxx.xx:9300]]]; nested: RemoteTransportException[[alex doe][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test][4] Phase[2] Execution failed]; nested: RemoteTransportException[[abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]]; nested: TranslogException[[test][4] failed to create new translog file]; nested: FileNotFoundException[/test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/4/translog/translog-15746574114492 (Permission denied)]; ]]


see above logs. does have any solution of above error?

Why are you running Elasticsearch 1.7? That's extremely well past EOL.

Never ever delete data directly from Elasticsearch data path as this can cause severe problems you can not rwcover from - always use the APIs.

1 Like

So now what can i do, so my cluster health is changed red to green?

I'd run:

curl -XDELETE http://localhost:9200/test

This should remove the faulty index.

If this does not work, you can stop the cluster, then run:

rm -rf /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test

And then restart the cluster.
This should work as far as I remember the 1.x version :wink:


[2021-09-20 16:21:45,521][DEBUG][action.bulk              ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2021-09-20 16:21:45,521][DEBUG][action.bulk              ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2021-09-20 16:21:45,521][DEBUG][action.bulk              ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2021-09-20 16:21:45,521][DEBUG][action.bulk              ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]

what is the solution of above error?

It does not look like an error. I can see it as DEBUG level.