# Courier Fetch: 3 out of 33 shards failed
**URL:** https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094
**Category:** Elasticsearch
**Created:** [March 1, 2018, 2:27pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094 "2018-03-01T14:27:53Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@hadi](https://discuss.elastic.co/u/hadi)
#### Post date: [March 1, 2018, 2:27pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/1 "2018-03-01T14:27:53Z")
Hello Everyone!
Once trying to check a specific index via Kibana, I get the following error!
The related logs that I see in my elastic search for this `Courier Fetch: 3 shard failed` are as follows:
```
2018-03-01T12:54:02,763][DEBUG][o.e.a.s.TransportSearchAction] [ip-10-50-30-150] [2120328] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ip-10-50-45-225][10.50.45.225:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [request_departure_date] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:301) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:165) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.search.DefaultSearchContext.getForField(DefaultSearchContext.java:501) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.search.fetch.subphase.DocValueFieldsFetchSubPhase.hitsExecute(DocValueFieldsFetchSubPhase.java:75) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:170) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:493) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.action.search.SearchTransportService$11.messageReceived(SearchTransportService.java:444) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.action.search.SearchTransportService$11.messageReceived(SearchTransportService.java:441) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1554) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.1.2.jar:6.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) ~[?:?]
at java.lang.Thread.run(Thread.java:844) [?:?]
[2018-03-01T12:54:02,782][DEBUG][o.e.a.s.TransportSearchAction] [ip-10-50-30-150] [2120322] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ip-10-50-45-225][10.50.45.225:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [request_departure_date] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
[2018-03-01T12:54:02,791][DEBUG][o.e.a.s.TransportSearchAction] [ip-10-50-30-150] [1909731] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ip-10-50-30-72][10.50.30.72:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [request_departure_date] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
```
And here is the part of my Elasticsearch config file, that I think is related to this issue:
```
http.enabled: true
thread_pool:
index:
queue_size: 10000
```
Please let me know how can I get rid of this error?
---
### Author:  [@jaisharma](https://discuss.elastic.co/u/jaisharma)
#### Post date: [March 1, 2018, 2:30pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/2 "2018-03-01T14:30:39Z")
What's the query you're trying?
---
### Author:  [@hadi](https://discuss.elastic.co/u/hadi)
#### Post date: [March 1, 2018, 2:36pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/3 "2018-03-01T14:36:57Z")
@jaisharma I don't use any query, I am just selecting a specific index and it returns me the following.
---
### Author:  [@jaisharma](https://discuss.elastic.co/u/jaisharma)
#### Post date: [March 1, 2018, 2:40pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/4 "2018-03-01T14:40:00Z")
Can you post the output of your [get mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html)
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 1, 2018, 2:41pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/5 "2018-03-01T14:41:27Z")
Are you in the discover page or on a visualisation or a dashboard?
Did you configure you Kibana index pattern to use `request_departure_date` field?
---
### Author:  [@hadi](https://discuss.elastic.co/u/hadi)
#### Post date: [March 1, 2018, 3:01pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/6 "2018-03-01T15:01:11Z")
```
GET /logstash-demo-all-*/_mapping
"request_date": {
"type": "date"
},
"request_departure_date": {
"type": "date"
},
"request_departure_station_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
```
---
### Author:  [@hadi](https://discuss.elastic.co/u/hadi)
#### Post date: [March 1, 2018, 3:05pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/7 "2018-03-01T15:05:37Z")
I am on the discover page.
And yes, `request_departure_date` is defined in the index patterns.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 1, 2018, 3:09pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/8 "2018-03-01T15:09:40Z")
Apparently, one of the shard "thinks" that `request_departure_date` is a `text` field.
Can't tell what happened but I think that the workaround would be most likely to reindex your data in another index which is correct.
---
### Author:  [@jaisharma](https://discuss.elastic.co/u/jaisharma)
#### Post date: [March 1, 2018, 3:16pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/9 "2018-03-01T15:16:59Z")
How can that happen ? I've never heard of difference in mapping across shards. Is it because of replication?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 1, 2018, 3:32pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/10 "2018-03-01T15:32:20Z")
It happened in very old versions but AFAIK it should not happen anymore in 6.x (may be 5.x) series anymore. I saw you are using 6.1 so that should not be the case.
Did you have any outage, something strange in logs when the index has been created?
---
### Author:  [@jaisharma](https://discuss.elastic.co/u/jaisharma)
#### Post date: [March 1, 2018, 4:04pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/11 "2018-03-01T16:04:51Z")
I'm running 2.4.0 and your statement about shards having different copy of mapping scares me.
How do it make sure i don't fall into this trap? Can i do something like querying with **preference=primary/replica** and check if i get consistent mappings? Shouldn't ES raise an alarm in case mappings are inconsistent across shards?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 1, 2018, 5:35pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/12 "2018-03-01T17:35:10Z")
@jaisharma In the past (can't remember exactly when from the top of my head), primary shards could get in parallel different kind of documents at exactly the same time. And if you are using only dynamic mapping and worse without any index template, then you can end up with the following.
ie. Let's say that primary shard 1 gets a first doc like:
```auto
{
"foo": "bar"
}
```
The primary shard 2 gets exactly at the same time something like:
```auto
{
"foo": "2017-12-23"
}
```
Primary one thinks it's a String.
Primary two thinks it's a Date.
And you have kind of inconsistent mapping here.
That happened in the past. This is no longer true with recent versions.
And that can not happen with index templates and explicit mapping even in old versions.
---
### Author:  [@hadi](https://discuss.elastic.co/u/hadi)
#### Post date: [March 1, 2018, 6:28pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/13 "2018-03-01T18:28:39Z")
Thank you @dadoonet.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 29, 2018, 6:28pm UTC](https://discuss.elastic.co/t/courier-fetch-3-out-of-33-shards-failed/122094/14 "2018-03-29T18:28:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.