# Getting Failed to execute phase \[query\], all shards failed **URL:** \ **Category:** Elasticsearch\ **Created:** [August 12, 2015, 9:19am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244 "2015-08-12T09:19:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 12, 2015, 9:19am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/1 "2015-08-12T09:19:49Z")
Hi All, Am getting the following exception Failed to execute phase [query], all shards failed org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:238) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:161) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100) at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63) at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51) at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:220) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:114) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) Can anyone help me out to handle this?????????.... ---
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 12, 2015, 9:34am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/2 "2015-08-12T09:34:11Z")
Even am getting the following exception while starting the ES.. ======================== translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [114]]; ]] [2015-08-12 04:29:04,872][WARN][index.engine] [Black Crow] [beta][3] failed to sync translog [2015-08-12 04:29:04,928][WARN][indices.cluster] [Black Crow] [[beta][3]] marking and sending shard failed due to [failed recovery] org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [beta][3] failed to recover shard at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:290) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:72) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:260) ... 4 more Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [115] at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:376) at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68) ... 5 more May I know the reason?????? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 13, 2015, 12:08am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/3 "2015-08-13T00:08:04Z")
> [@anusha6](#): > > translog corruption while reading from stream at Something has gone wrong and your translog is corrupted. You'll have to find it for that shard and probably delete it. ---
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 13, 2015, 6:00am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/4 "2015-08-13T06:00:05Z")
Hi Mark, Thanks for your reply.... So is this not because of huge data? Here is the information regarding shards: logstash-2014.12.09 4 p STARTED 11230 2.6mb 172.16.112.25 Shooting Star logstash-2014.12.09 4 r UNASSIGNED logstash-2014.12.09 0 p STARTED 11410 2.6mb 172.16.112.25 Shooting Star logstash-2014.12.09 0 r UNASSIGNED logstash-2014.12.09 3 p STARTED 11263 2.6mb 172.16.112.25 Shooting Star logstash-2014.12.09 3 r UNASSIGNED logstash-2014.12.09 1 p STARTED 11113 2.5mb 172.16.112.25 Shooting Star logstash-2014.12.09 1 r UNASSIGNED logstash-2014.12.09 2 p STARTED 11459 2.7mb 172.16.112.25 Shooting Star logstash-2014.12.09 2 r UNASSIGNED logstash-2014.12.08 2 p STARTED 8959 2.1mb 172.16.112.25 Shooting Star logstash-2014.12.08 2 r UNASSIGNED logstash-2014.12.08 0 p STARTED 8730 2.1mb 172.16.112.25 Shooting Star logstash-2014.12.08 0 r UNASSIGNED logstash-2014.12.08 3 p STARTED 8861 2.1mb 172.16.112.25 Shooting Star logstash-2014.12.08 3 r UNASSIGNED logstash-2014.12.08 1 p STARTED 8843 2.1mb 172.16.112.25 Shooting Star logstash-2014.12.08 1 r UNASSIGNED logstash-2014.12.08 4 p STARTED 8915 2.1mb 172.16.112.25 Shooting Star logstash-2014.12.08 4 r UNASSIGNED beta 4 p STARTED 31567980 3gb 172.16.112.25 Shooting Star beta 4 r UNASSIGNED beta 0 p STARTED 31493375 3gb 172.16.112.25 Shooting Star beta 0 r UNASSIGNED beta 3 p STARTED 31568715 3gb 172.16.112.25 Shooting Star beta 3 r UNASSIGNED beta 1 p STARTED 31567359 3gb 172.16.112.25 Shooting Star beta 1 r UNASSIGNED beta 2 p STARTED 31567649 3gb 172.16.112.25 Shooting Star beta 2 r UNASSIGNED prod 2 p STARTED 25318076 4.3gb 172.16.112.25 Shooting Star prod 2 r UNASSIGNED **prod 0 p UNASSIGNED** **prod 0 r UNASSIGNED** prod 3 p STARTED 25302767 4.3gb 172.16.112.25 Shooting Star prod 3 r UNASSIGNED prod 1 p STARTED 25316553 4.3gb 172.16.112.25 Shooting Star prod 1 r UNASSIGNED prod 4 p STARTED 25318895 4.3gb 172.16.112.25 Shooting Star prod 4 r UNASSIGNED 1. Where among them prod [0] shard is **unassigned** it is showing, Isn't it because of the more memory usage.??. 2. If not can you guide me how to delete a shard data as you said???? 3. And does the elasticsearch manage the data to be delete, when it occupies a huge memory space???? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 13, 2015, 6:07am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/5 "2015-08-13T06:07:03Z")
It might be easier if you just reindex the data in `beta`, are you able to do that? The logs don't seem to show why the `prod` shard is not being assigned though. It also doesn't look like you have too much data, but then I don't know what heap you have. You need to manage data in ES, it's up to you to figure out when things need to be deleted. [Elasticsearch Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) does automate it to a degree, but you still need to make decisions. ---
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 13, 2015, 6:26am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/6 "2015-08-13T06:26:46Z")
As the corruption is happened in prod index , why do I have to re-index the beta data? And for re-indexing I don't have backup data.. Is there any way to delete only a particular shard data? And My machine is having heap memory of '40GB'..... Is there any way to fix these type of issues at starting stage????? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 13, 2015, 6:46am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/7 "2015-08-13T06:46:36Z")
If you have `ES_HEAP_SIZE` (is java min and max) set to 40GG, that's too much. You should keep it to 30.5Gb or less. There's no way to fix this at startup as something bad has happened and made the data corrupt. And if you delete the shard you lose the data in it. **NOTE** - the following is done entirely at your own risk. If you don't have the data elsewhere, or a backup, there is a chance you will lose data. MAKE SURE YOU HAVE A BACKUP. Figure out where your [ES data directory is](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-dir-layout.html). Stop Elasticsearch and find the directory holding shard 3 of the `beta` index and then move the translog file to somewhere else. Then restart ES. That will at least clear your translog corruption issue. As I said though, you will likely lose whatever data is in the translog. ---
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 13, 2015, 7:05am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/8 "2015-08-13T07:05:22Z")
May I know why that I have to do > [@warkolm](#): > > shard 3 of the beta index and then move the translog file to somewhere else As I have data corruption in prod [0] shard ???? And also where can I find the 'ES\_HEAP\_SIZE' to set?? And if I set the size to that extent, after my data exceeds that limit of 30.5gb what will happen ???? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 13, 2015, 7:26am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/9 "2015-08-13T07:26:38Z")
> [@anusha6](#): > > As I have data corruption in prod [0] shard ???? Well the log you posted said shard 3 of the `beta` index, hence my comment. Elasticsearch doesn't store everything in memory, it just puts certain things in there as needed (that's a gross simplification, but it seems like you are still learning). Setting the heap size will depend on how you installed Elasticsearch to begin with. ---
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 13, 2015, 9:30am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/10 "2015-08-13T09:30:08Z")
Yes Thanks Mart, the logs that I got next time having prod [0], so based on that I said, Mark I just copied few shards in new index. But when I see the mappings of that new index it is not showing any mappings.. These mappings I created for new index.. PUT facebook3 { "mappings": { "face": {"\_ttl": {"enabled": true, "default": "3m"}} } } Inorder to remove the data for every 3 mins... And I copied an index called facebook shards in this new index by stopping elasticsearch, after i copied those shards I again started ES, When I run the search query am getting records: GET facebook3/\_search { "\_index": "facebook3", "\_type": "face", "\_id": "1", "\_score": 1, "\_source": { "date": "2015-03-31 13:12:13", "name": "santosh" } } ] But when I have seen the mappings, no mappings created for type face. May I know the reason???? ---
**Author:** ![anusha6](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@anusha6](https://discuss.elastic.co/u/anusha6)\ **Post date:** [August 13, 2015, 10:20am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/11 "2015-08-13T10:20:52Z")
Hi Mark Thanks for your support, I finally took the backup and done the data deletion at specific time using \_ttl ... Resolved the issue... 😃 ---
**Author:** ![imsonujangra](https://avatars.discourse-cdn.com/v4/letter/i/3e96dc/32.png) [@imsonujangra](https://discuss.elastic.co/u/imsonujangra)\ **Post date:** [January 2, 2017, 8:07am UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/12 "2017-01-02T08:07:27Z")
I'm also getting this log for my elasticsearch. [LiveUsedcarES-03] All shards failed for phase: [query] org.elasticsearch.transport.RemoteTransportException: [LiveUsedcarES-03][inet[/174.18.11.233:9201]][indices:data/read/search[phase/query]] Caused by: org.elasticsearch.search.SearchParseException: [usedcar\_index\_1312][5]: query[filtered(+is\_premium:0)-\>BooleanFilter(+cache(price:[10000 TO \*]) +cache(is\_gaadi:^@^@^@^A))],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"filtered":{"filter":{"bool":{"must":[{"range":{"price":{"gte":10000}}},{"term":{"is\_gaadi":"1"}}]}},"query":{"bool":{"must":[{"term":{"is\_premium":0}}]}}}},"aggs":{"max\_price":{"max":{"field":"price"}},"min\_price":{"min":{"field":"price"}},"max\_kms\_run":{"max":{"field":"kms\_run"}},"min\_kms\_run":{"min":{"field":"kms\_run"}},"max\_registration\_year":{"max":{"field":"registration\_year"}},"min\_registration\_year":{"min":{"field":"registration\_year"}}},"post\_filter":{"bool":{"must":[{"terms":{"listing\_dealer\_info":"dealer"}},{"term":{"is\_gaadi":"1"}}],"must\_not":[{"ids":{"values":["1164950","1170049","1258563","1281903","1181971","1276341","1206654","1218051","1205663","1223747","1253233","1207243","1222722","1192937","1199283","1191936","1165433","1256900","1203704","1184709","1258558","1275804","1275817","1204879","1180492","1223042","1190552","1219339","1203808","1198643","1258602","1202618","1277314","1271670","1217135","1218234","1205634","1269051","1213430","1279729","1224704","1219012","1269616","1214183","1180846","1275610","1218055","1214480","1276472","1178313","1276451","1276484","1263642","1180872","1225433","1200295","1194396","1192859","1222724","1189660","1217582","1219916","1225050","1280365","1219404","1276196","1277769","1276462","1204706","1173916","1176627","1189071","1210921","1220188","1278145","1176883","1204881","1200495","1276326","1276439","1170984","1186233","1273457","1222999","1169833","1252857","1164109","1269541","1277609","1276424","1218608","1201991","1225191","1227126","1276354","1275559","1209934","1278163","1216924","1223968","1214033","1263647","1191590","1279307","1276288","1222131","1216899","1276315","1195211","1276404","1252944","1276189","1276429","1276481","1200773","1208118","1258716","1276422","1276280","1273343","1216098","1176302","1219127","1147758","1209064","1165347","1193310","1203248","1213901","1276302","1276441","1222882","1261852","1280929","1218288","1272117","1276327","1164785","1223513","1262280","1223800","1277209","1182312","1185943","1204050","1276485","1211192","1258437","1216261","1269825","1223735","1219442","1194434","1186332","1214215","1276471","1211119","1278938","1272139","1157684","1199855","1194580","1253830","1276227","1280720","1276447","1281696","1200926","1281006","1223729","1189593","1198742","1200176","1269816","1217401","1258261","1209061","1263651","1200040","1159183","1217894"]}}]}},"size":0,"sort":{"goodness\_score":{"order":"desc"}},"from":0}]] Can you please tell where i'm going wrong. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 2, 2017, 8:42pm UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/13 "2017-01-02T20:42:51Z")
Please start your own thread 🙂 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 2, 2017, 8:42pm UTC](https://discuss.elastic.co/t/getting-failed-to-execute-phase-query-all-shards-failed/27244/14 "2017-01-02T20:42:59Z")