Error=failed to build store metadata. checking segment info integrity (with commit [n
I have removed the older data directly(index) from the node data directory, while i have restart the elasticsearch server and see in the log i'm getting this above error. so what is the solution of above error.
Welcome to our community! 
Please provide more of the error/log that you are seeing.
What do you mean here, you delete the data directory from the disk?
yes remove old index in elasticsearch. whenever i try to start elasticsearch service using sudo systemctl start elasticsearch.service is not start. why ?
You should never delete data from the disk, you should always use the APIs - Delete index API | Elasticsearch Guide [7.14] | Elastic
What does the full Elasticsearch log show from startup to error?
That doesn't show any errors.
Also please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you 
I have a ES cluster showing the status as red.
{ "cluster_name" : "XYZ", "status" : "red", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 72, "active_shards" : 72, "relocating_shards" : 0, "initializing_shards" : 2, "unassigned_shards" : 78, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0 }
I have restart the server because of low disk watermark error, and delete some index through a winscp tool, a index is older like 2019 & 2020. after the restart the server i got above error and metrics is not stored in es. also i'm using apiman and in apiman error log i see adding metrics to es error. most of shard is UNASSIGNED state pls provide the solution. anyone have a solution of above error?
You really need to show these errors please.
Below is the APIMAN logs
Error adding metric to ES:
ERROR [stderr] (ESMetricsConsumer) com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22 path $
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.JsonParser.parse(JsonParser.java:65)
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.JsonParser.parse(JsonParser.java:45)
ERROR [stderr] (ESMetricsConsumer) at io.searchbox.action.AbstractAction.parseResponseBody(AbstractAction.java:96)
ERROR [stderr] (ESMetricsConsumer) at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:130)
ERROR [stderr] (ESMetricsConsumer) at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:125)
ERROR [stderr] (ESMetricsConsumer) at io.searchbox.core.Bulk.createNewElasticSearchResult(Bulk.java:30)
ERROR [stderr] (ESMetricsConsumer) at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:146)
ERROR [stderr] (ESMetricsConsumer) at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:65)
ERROR [stderr] (ESMetricsConsumer) at io.apiman.gateway.engine.es.ESMetrics.processQueue(ESMetrics.java:126)
ERROR [stderr] (ESMetricsConsumer) at io.apiman.gateway.engine.es.ESMetrics$1.run(ESMetrics.java:100)
ERROR [stderr] (ESMetricsConsumer) at java.lang.Thread.run(Thread.java:745)
ERROR [stderr] (ESMetricsConsumer) Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22 path $
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1573)
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1423)
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:546)
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.stream.JsonReader.peek(JsonReader.java:429)
ERROR [stderr] (ESMetricsConsumer) at com.google.gson.JsonParser.parse(JsonParser.java:60)
ERROR [stderr] (ESMetricsConsumer) ... 10 more
below is the elasticsearch server logs
[WARN ][indices.cluster ] [abc Two] [[test][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [test][1]: Recovery failed from [alex doe][XB0zayb3R5CCIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8QdW3ed-ziSqhpw][XYZ.test.in][inet[xx.x.xxx.xx:9300]]
at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280)
at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70)
at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:567)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: [alex doe][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [test][1] Phase[2] Execution failed
at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:902)
at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:780)
at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125)
at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: [abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]
Caused by: org.elasticsearch.index.translog.TranslogException: [test][1] failed to create new translog file
at org.elasticsearch.index.translog.fs.FsTranslog.newTranslog(FsTranslog.java:253)
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:189)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:146)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1355)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1350)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:870)
at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:307)
at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:290)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/1/translog/translog-15784574114474 (Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.elasticsearch.index.translog.fs.RafReference.<init>(RafReference.java:49)
at org.elasticsearch.index.translog.fs.FsTranslog.newTranslog(FsTranslog.java:251)
... 13 more
Below is the clone of elasticseach server log
[WARN ][cluster.action.shard ] [alex doe] [test][4] received shard failed for [test][4], node[gywAkbl8QdW3ed-ziSqhpw], [R], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-16], details[shard failure [failed recovery][RecoveryFailedException[[test][4]: Recovery failed from [alex doe][XB0zayb3RCCIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8QdW3ed-ziSqhpw][XYZ.test.in][inet[/xx.x.xxx.xx:9300]]]; nested: RemoteTransportException[[Baron Macabre][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test][4] Phase[2] Execution failed]; nested: RemoteTransportException[[abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]]; nested: TranslogException[[test][4] failed to create new translog file]; nested: FileNotFoundException[/test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/4/translog/translog-1574574114492 (Permission denied)]; ]]], indexUUID [rAHH1oQQygkCB2bdzqZQ], reason [shard failure [failed recovery][RecoveryFailedException[[test][4]: Recovery failed from [alex doe][XB0zayb3RC5CIlFK_RMBOpA][XYZ.test.in][inet[/xx.x.xxx.xx:9300]] into [abc Two][gywAkbl8Qd7W3ed-ziSqhpw][XYZ.test.in][inet[/xx.x.xxx.xx:9300]]]; nested: RemoteTransportException[[alex doe][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test][4] Phase[2] Execution failed]; nested: RemoteTransportException[[abc Two][inet[/xx.x.xxx.xx:9300]][internal:index/shard/recovery/prepare_translog]]; nested: TranslogException[[test][4] failed to create new translog file]; nested: FileNotFoundException[/test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test/4/translog/translog-15746574114492 (Permission denied)]; ]]
see above logs. does have any solution of above error?
Never ever delete data directly from Elasticsearch data path as this can cause severe problems you can not rwcover from - always use the APIs.
So now what can i do, so my cluster health is changed red to green?
I'd run:
curl -XDELETE http://localhost:9200/test
This should remove the faulty index.
If this does not work, you can stop the cluster, then run:
rm -rf /test1/def/elasticsearch-1.7.2/data/apiman/nodes/0/indices/test
And then restart the cluster.
This should work as far as I remember the 1.x version 
[2021-09-20 16:21:45,521][DEBUG][action.bulk ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2021-09-20 16:21:45,521][DEBUG][action.bulk ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2021-09-20 16:21:45,521][DEBUG][action.bulk ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2021-09-20 16:21:45,521][DEBUG][action.bulk ] [xyz] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
what is the solution of above error?
It does not look like an error. I can see it as DEBUG level.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.