# Data corruption in a few elasticsearch indices
**URL:** https://discuss.elastic.co/t/data-corruption-in-a-few-elasticsearch-indices/104824
**Category:** Elasticsearch
**Created:** [October 23, 2017, 12:49am UTC](https://discuss.elastic.co/t/data-corruption-in-a-few-elasticsearch-indices/104824 "2017-10-23T00:49:38Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@amiexc](https://discuss.elastic.co/u/amiexc)
#### Post date: [October 23, 2017, 12:49am UTC](https://discuss.elastic.co/t/data-corruption-in-a-few-elasticsearch-indices/104824/1 "2017-10-23T00:49:38Z")
Hi all,
We've encountered a data corruption in our main production elasticsearch instance. We started getting the following response when querying the corrupted index:
{"error":{"root\_cause":[{"type":"no\_shard\_available\_action\_exception","reason":"No shard available for [get [hotels3][points][4185]: routing [null]]"}],"type":"no\_shard\_available\_action\_exception","reason":"No shard available for [get [hotels3][points][4185]: routing [null]]"},"status":503}
\_cat returns the following state (we didn't change any settings):
hotels3 0 p UNASSIGNED
hotels3 0 r UNASSIGNED
replicated cluster instances got populated with the same error and lost all data which was also a big hit.
restarting elasticsearch did not recover the indices.
I saw many topics with the same error however none had a reason that applied to us or a solution that worked for us.
1. what could cause this issue and how to prevent it
2. how to fix the indices
3. how to prevent the replicated cluster nodes from getting corrupted as well?
Any help will be greatly appreciated.
Thank you,
Ami
cluster logs:
[2017-10-23T09:57:22,773][DEBUG][o.e.a.s.TransportSearchAction] [escl01] All shards failed for phase: [query]
org.elasticsearch.action.NoShardAvailableActionException: null
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:122) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:240) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:146) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:67) ~
...
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_121]
[2017-10-23T09:57:22,777][WARN][r.suppressed] path: /hotels3/\_count, params: {index=hotels3}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onInitialPhaseResult(AbstractSearchAsyncAction.java:223) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:122) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:240) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:146) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:67) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.3.0.jar:5.3.0]
...
...
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_121]
Caused by: org.elasticsearch.action.NoShardAvailableActionException
... 58 more
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [October 23, 2017, 8:26am UTC](https://discuss.elastic.co/t/data-corruption-in-a-few-elasticsearch-indices/104824/2 "2017-10-23T08:26:48Z")
You need to search the logs of all your nodes for this particular shard, as this shard is not part of your cluster anymore.
Have you maybe lost nodes, that held that shard?
---
### Author:  [@amiexc](https://discuss.elastic.co/u/amiexc)
#### Post date: [October 24, 2017, 1:37am UTC](https://discuss.elastic.co/t/data-corruption-in-a-few-elasticsearch-indices/104824/3 "2017-10-24T01:37:12Z")
Thanks for your reply. The index is configured with number of shards: 1 and the other nodes are replicas of the main one (main one for us in the sense that it's the only node we're pushing data to, the other nodes are used for read only).
In the 2 other nodes I found this error:
node2:
[internal:index/shard/recovery/start\_recovery] sent error response
742720:org.elasticsearch.indices.recovery.DelayRecoveryException: source node does not have the shard listed in its state as allocated on the node
node3:
729465-[2017-10-18T20:10:22,074][TRACE][o.e.t.T.tracer] [escl03] [14671845][internal:index/shard/recovery/start\_recovery] received request
729466-[2017-10-18T20:10:22,075][TRACE][o.e.t.T.tracer] [escl03] [14671845][internal:index/shard/recovery/start\_recovery] sent error response
729467:org.elasticsearch.index.shard.ShardNotFoundException: no such shard
729468- at org.elasticsearch.index.IndexService.getShard(IndexService.java:208) ~[elasticsearch-5.3.0.jar:5.3.0]
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 21, 2017, 1:37am UTC](https://discuss.elastic.co/t/data-corruption-in-a-few-elasticsearch-indices/104824/4 "2017-11-21T01:37:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.