# N of N shards failed on Kibana after upgrading from 6.3.2 to 6.4.2

**URL:** <https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266>\
**Category:** Elasticsearch\
**Created:** [November 3, 2018, 5:20pm UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266 "2018-11-03T17:20:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [November 3, 2018, 5:20pm UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266/1 "2018-11-03T17:20:05Z")

</div>

Hello there,

We recently upgraded our ELK from 6.3.2 to 6.4.2 and I noticed many of the dashboards started having errors like "N of M shards failed".  
Also in the elasticsearch log, I saw the following:

==============  
[2018-11-03T12:05:15,657][DEBUG][o.e.a.s.TransportSearchAction] [Elastinode03] [metricbeat-6.3.2-2018.11.03][2], node[9BK8ywvQRwyKT90OXGddww], [P], s[STARTED], a[id=l5wMagQhQnSD7-rrosofPQ]  
: Failed to execute [SearchRequest{searchType=QUERY\_THEN\_FETCH, indices=[metricbeat-6.4.1-2018.10.01, metricbeat-6.4.1-2018.10.02, metricbeat-6.4.1-2018.10.03, metricbeat-6.3.2-2018.09.20,  
metricbeat-6.3.2-2018.10.09, metricbeat-6.4.1-2018.09.19, metricbeat-6.3.2-2018.10.08, metricbeat-6.4.2-2018.10.29, metricbeat-6.3.2-2018.10.07, metricbeat-6.3.2-2018.10.06, metricbeat-6.3.  
2-2018.10.05... (almost all metribeat 6.3.2 indices)...  
indicesOptions=IndicesOptions[ignore\_unavailable=true, allow\_no\_indices=true, expand\_wildcards\_open=true, expand\_wildcards  
\_closed=false, allow\_aliases\_to\_multiple\_indices=true, forbid\_closed\_indices=true, ignore\_aliases=false], types=, routing='null', preference='1541264530052', requestCache=null, scroll=nul  
l, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=64, allowPartialSearchResults=true, source={ many fields...)...  
"aggregations":{"1":{"avg":{"field":"system.cpu.user.pct"}}}}}}}}}] lastShard [true]  
org.elasticsearch.transport.RemoteTransportException: [hlsoelse1a-02][10.100.35.177:9300][indices:data/read/search[phase/query]]  
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [beat.name] in order to load fielddata in memory by uninverting the inv  
erted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Is there anything we should do to fix this?

Thanks a lot in advance

Li

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [November 3, 2018, 5:31pm UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266/2 "2018-11-03T17:31:12Z")

</div>

This is output on the same elasticnode:

curl -X GET "[https://elasticnode3:9200/\_cluster/stats?human&pretty](https://elasticnode3:9200/_cluster/stats?human&pretty)" -k  
{  
"\_nodes" : {  
"total" : 3,  
"successful" : 3,  
"failed" : 0  
},  
"cluster\_name" : "hlsmtc",  
"timestamp" : 1541266150537,  
"status" : "green",  
"indices" : {  
"count" : 707,  
"shards" : {  
"total" : 4311,  
"primaries" : 2155,  
"replication" : 1.0004640371229698,  
"index" : {  
"shards" : {  
"min" : 2,  
"max" : 10,  
"avg" : 6.097595473833097  
},  
"primaries" : {  
"min" : 1,  
"max" : 5,  
"avg" : 3.048090523338048  
},  
"replication" : {  
"min" : 1.0,  
"max" : 2.0,  
"avg" : 1.0014144271570014  
}  
}  
},  
"docs" : {  
"count" : 1706884793,  
"deleted" : 1118462  
},  
"store" : {  
"size" : "636.9gb",  
"size\_in\_bytes" : 683960705355  
},  
"fielddata" : {  
"memory\_size" : "6.3mb",  
"memory\_size\_in\_bytes" : 6641160,  
"evictions" : 0  
},  
"query\_cache" : {  
"memory\_size" : "328.7mb",  
"memory\_size\_in\_bytes" : 344737278,  
"total\_count" : 12051049,  
"hit\_count" : 189111,  
"miss\_count" : 11861938,  
"cache\_size" : 3855,  
"cache\_count" : 6781,  
"evictions" : 2926  
},  
"completion" : {  
"size" : "0b",  
"size\_in\_bytes" : 0  
},  
"segments" : {  
"count" : 30826,  
"memory" : "2.2gb",  
"memory\_in\_bytes" : 2422690768,  
"terms\_memory" : "1.6gb",  
"terms\_memory\_in\_bytes" : 1801506514,  
"stored\_fields\_memory" : "352.5mb",  
"stored\_fields\_memory\_in\_bytes" : 369670176,  
"term\_vectors\_memory" : "0b",  
"term\_vectors\_memory\_in\_bytes" : 0,  
"norms\_memory" : "29.4mb",  
"norms\_memory\_in\_bytes" : 30908480,  
"points\_memory" : "103.5mb",  
"points\_memory\_in\_bytes" : 108629958,  
"doc\_values\_memory" : "106.7mb",  
"doc\_values\_memory\_in\_bytes" : 111975640,  
"index\_writer\_memory" : "95.9mb",  
"index\_writer\_memory\_in\_bytes" : 100587218,  
"version\_map\_memory" : "21.1mb",  
"version\_map\_memory\_in\_bytes" : 22133189,  
"fixed\_bit\_set" : "1mb",  
"fixed\_bit\_set\_memory\_in\_bytes" : 1085256,  
"max\_unsafe\_auto\_id\_timestamp" : 1541231846997,  
"file\_sizes" : { }  
}  
},  
"nodes" : {  
"count" : {  
"total" : 3,  
"data" : 3,  
"coordinating\_only" : 0,  
"master" : 3,  
"ingest" : 3  
},  
"versions" : [  
"6.4.2"  
],  
"os" : {  
"available\_processors" : 12,  
"allocated\_processors" : 12,  
"names" : [  
{  
"name" : "Linux",  
"count" : 3  
}  
],  
"mem" : {  
"total" : "46.5gb",  
"total\_in\_bytes" : 49968709632,  
"free" : "508.5mb",  
"free\_in\_bytes" : 533209088,  
"used" : "46gb",  
"used\_in\_bytes" : 49435500544,  
"free\_percent" : 1,  
"used\_percent" : 99  
}  
},  
"process" : {  
"cpu" : {  
"percent" : 48  
},  
"open\_file\_descriptors" : {  
"min" : 9784,  
"max" : 25861,  
"avg" : 19880  
}  
},  
"jvm" : {  
"max\_uptime" : "2.7d",  
"max\_uptime\_in\_millis" : 237214523,  
"versions" : [  
{  
"version" : "1.8.0\_172",  
"vm\_name" : "Java HotSpot(TM) 64-Bit Server VM",  
"vm\_version" : "25.172-b11",  
"vm\_vendor" : "Oracle Corporation",  
"count" : 1  
},  
{  
"version" : "1.8.0\_191",  
"vm\_name" : "OpenJDK 64-Bit Server VM",  
"vm\_version" : "25.191-b12",  
"vm\_vendor" : "Oracle Corporation",  
"count" : 2  
}  
],  
"mem" : {  
"heap\_used" : "12.8gb",  
"heap\_used\_in\_bytes" : 13752094352,  
"heap\_max" : "23.9gb",  
"heap\_max\_in\_bytes" : 25665208320  
},  
"threads" : 463  
},  
"fs" : {  
"total" : "1.4tb",  
"total\_in\_bytes" : 1610455449600,  
"free" : "843.4gb",  
"free\_in\_bytes" : 905666670592,  
"available" : "843.4gb",  
"available\_in\_bytes" : 905666670592  
},  
"plugins" : [  
{  
"name" : "ingest-geoip",  
"version" : "6.4.2",  
"elasticsearch\_version" : "6.4.2",  
"java\_version" : "1.8",  
"description" : "Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database",  
"classname" : "org.elasticsearch.ingest.geoip.IngestGeoIpPlugin",  
"extended\_plugins" : ,  
"has\_native\_controller" : false  
},  
{  
"name" : "discovery-ec2",  
"version" : "6.4.2",  
"elasticsearch\_version" : "6.4.2",  
"java\_version" : "1.8",  
"description" : "The EC2 discovery plugin allows to use AWS API for the unicast discovery mechanism.",  
"classname" : "org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin",  
"extended\_plugins" : ,  
"has\_native\_controller" : false  
},  
{  
"name" : "repository-s3",  
"version" : "6.4.2",  
"elasticsearch\_version" : "6.4.2",  
"java\_version" : "1.8",  
"description" : "The S3 repository plugin adds S3 repositories",  
"classname" : "org.elasticsearch.repositories.s3.S3RepositoryPlugin",  
"extended\_plugins" : ,  
"has\_native\_controller" : false  
}  
],  
"network\_types" : {  
"transport\_types" : {  
"security4" : 3  
},  
"http\_types" : {  
"security4" : 3  
}  
}  
}  
}

=============  
I ran the following query and got {}.... what did this mean?  
curl -XGET '[https://elasticnode3:9200/\_template/metricbeat?pretty=true](https://elasticnode3:9200/_template/metricbeat?pretty=true)' -k  
{ }

This is very important, after upgrade, almost all of our dashboards stopped working, they are all the default dashboards from V6.3.2...

Please help. Thanks

Li

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [November 5, 2018, 5:38am UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266/3 "2018-11-05T05:38:48Z")

</div>

curl -X PUT "[https://elasticnode03:9200/metricbeat-\*/\_mapping/\_doc](https://elasticnode03:9200/metricbeat-*/_mapping/_doc)" -H 'Content-Type: application/json' -d'  
{  
"properties": {  
"beat.name": {  
"type": "text",  
"fielddata": true  
}  
}  
}  
'

On all Elasticnodes... and got:

{"error":{"root\_cause":[{"type":"remote\_transport\_exception","reason":"[elasticnode01][xx.xxx.xx.xxx:9300 [indices:admin/mapping/put]"}],"type":"illegal\_argument\_exception","reason":"Rejecting mapping update to [metricbeat-6.4.2-2018.10.31] as the final mapping would have more than 1 type: [\_doc, doc]"},"status":400}

Still have the same errors...

And the similar errors occurred on all index patterns, filebeat-_, packetbeat-_....  
All we did was to upgraded from 6.3.2 to 6.4.2... nothing else was changed but after upgrade, almost all dashboards started as various errors..

Please help and thanks

Li

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [November 7, 2018, 3:22pm UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266/4 "2018-11-07T15:22:30Z")

</div>

Any updates on this please?

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [November 7, 2018, 4:05pm UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266/5 "2018-11-07T16:05:56Z")

</div>

I don't think enabling fielddata is the solution here. The bigger issue is that the `beat.name` field has been mapped as a `text` field rather than a `keyword` field in your indexes. This suggests that the [Metricbeat index template](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-template.html) has not been loaded.

Does your Metricbeat have `setup.template.enabled` set to `false`? If so, you need to manually [load the index template](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-template.html#load-template-manually):

```auto
metricbeat setup --template

```

Note that this will only fix future indexes. Any existing metricbeat indexes would have to be [reindexed](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2018, 4:05pm UTC](https://discuss.elastic.co/t/n-of-n-shards-failed-on-kibana-after-upgrading-from-6-3-2-to-6-4-2/155266/6 "2018-12-05T16:05:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.