# Shards Failure - Kibana Dashboard **URL:** \ **Category:** Kibana\ **Created:** [November 7, 2022, 4:32am UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287 "2022-11-07T04:32:19Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![balaa](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@balaa](https://discuss.elastic.co/u/balaa)\ **Post date:** [November 7, 2022, 4:32am UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/1 "2022-11-07T04:32:19Z")
Hi Team, We are are using ELK 7.16.3 version and when ever the timeframe is more than 30 days i am getting the shards failure in my dashboard. ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/9/194656bbcaf4ff041d2867bdc09023eea68c04d7.png) **node\_disconnected\_exception at** **shard** **0** **index** **sd-daily-20220422-00105-000004** **node** **ODanHuRARUSZsFk2QU7wtw** **Type** node\_disconnected\_exception **Reason** [elk-master-0][172.16.20.10:9300][indices:data/read/search[phase/query]] disconnected Response code { "took": 5073, "timed\_out": false, "\_shards": { "total": 3, "successful": 2, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "sd-daily -20220422-00105-000004", "node": "ODanHuRARUSZsFk2QU7wtw", "reason": { "type": "node\_disconnected\_exception", "reason": "[elk-master-0][172.16.20.10:9300][indices:data/read/search[phase/query]] disconnected" } } ] }, ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 7, 2022, 5:31am UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/2 "2022-11-07T05:31:54Z")
What is the output from the [`_cluster/stats?pretty&human` API](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/cluster-stats.html)? ---
**Author:** ![balaa](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@balaa](https://discuss.elastic.co/u/balaa)\ **Post date:** [November 7, 2022, 6:26am UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/3 "2022-11-07T06:26:33Z")
{ "\_nodes" : { "total" : 3, "successful" : 3, "failed" : 0 }, "cluster\_name" : "elk-gd", "cluster\_uuid" : "f0hXcMLaQo-RlnM1zJrRqQ", "timestamp" : 1667801136032, "status" : "green", "indices" : { "count" : 1673, "shards" : { "total" : 2487, "primaries" : 2331, "replication" : 0.06692406692406692, "index" : { "shards" : { "min" : 1, "max" : 6, "avg" : 1.4865511057979677 }, "primaries" : { "min" : 1, "max" : 3, "avg" : 1.393305439330544 }, "replication" : { "min" : 0.0, "max" : 1.0, "avg" : 0.08846383741781232 } } }, "docs" : { "count" : 1774970083, "deleted" : 4479637 }, "store" : { "size" : "572.4gb", "size\_in\_bytes" : 614621542072, "total\_data\_set\_size" : "572.4gb", "total\_data\_set\_size\_in\_bytes" : 614621542072, "reserved" : "0b", "reserved\_in\_bytes" : 0 }, "fielddata" : { "memory\_size" : "1mb", "memory\_size\_in\_bytes" : 1132944, "evictions" : 0 }, "query\_cache" : { "memory\_size" : "12.8mb", "memory\_size\_in\_bytes" : 13439539, "total\_count" : 6754554, "hit\_count" : 210691, "miss\_count" : 6543863, "cache\_size" : 3237, "cache\_count" : 10458, "evictions" : 7221 }, "completion" : { "size" : "0b", "size\_in\_bytes" : 0 }, "segments" : { "count" : 17862, "memory" : "239.1mb", "memory\_in\_bytes" : 250772330, "terms\_memory" : "167.5mb", "terms\_memory\_in\_bytes" : 175647856, "stored\_fields\_memory" : "10mb", "stored\_fields\_memory\_in\_bytes" : 10514464, "term\_vectors\_memory" : "0b", "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory" : "11.2mb", "norms\_memory\_in\_bytes" : 11809344, "points\_memory" : "0b", "points\_memory\_in\_bytes" : 0, "doc\_values\_memory" : "50.3mb", "doc\_values\_memory\_in\_bytes" : 52800666, "index\_writer\_memory" : "1.1gb", "index\_writer\_memory\_in\_bytes" : 1198878944, "version\_map\_memory" : "21.9mb", "version\_map\_memory\_in\_bytes" : 23008231, "fixed\_bit\_set" : "13.7mb", "fixed\_bit\_set\_memory\_in\_bytes" : 14374768, "max\_unsafe\_auto\_id\_timestamp" : 1667800451011, "file\_sizes" : { } }, "mappings" : { "field\_types" : [ { "name" : "alias", "count" : 3092, "index\_count" : 1355, "script\_count" : 0 }, { "name" : "binary", "count" : 5, "index\_count" : 5, "script\_count" : 0 }, { "name" : "boolean", "count" : 41385, "index\_count" : 1522, "script\_count" : 0 }, { "name" : "byte", "count" : 1316, "index\_count" : 1316, "script\_count" : 0 }, { "name" : "constant\_keyword", "count" : 2663, "index\_count" : 888, "script\_count" : 0 }, { "name" : "date", "count" : 66455, "index\_count" : 1643, "script\_count" : 0 }, { "name" : "date\_nanos", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "date\_range", "count" : 5, "index\_count" : 5, "script\_count" : 0 }, { "name" : "double", "count" : 5260, "index\_count" : 66, "script\_count" : 0 }, { "name" : "double\_range", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "flattened", "count" : 10894, "index\_count" : 922, "script\_count" : 0 }, { "name" : "float", "count" : 19877, "index\_count" : 1396, "script\_count" : 0 }, { "name" : "float\_range", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "geo\_point", "count" : 11296, "index\_count" : 1357, "script\_count" : 0 }, { "name" : "geo\_shape", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "half\_float", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "histogram", "count" : 1279, "index\_count" : 1279, "script\_count" : 0 }, { "name" : "integer", "count" : 295, "index\_count" : 99, "script\_count" : 0 }, { "name" : "integer\_range", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "ip", "count" : 24352, "index\_count" : 1358, "script\_count" : 0 }, { "name" : "ip\_range", "count" : 5, "index\_count" : 5, "script\_count" : 0 }, { "name" : "keyword", "count" : 1335162, "index\_count" : 1643, "script\_count" : 0 }, { "name" : "long", "count" : 296206, "index\_count" : 1573, "script\_count" : 0 }, { "name" : "long\_range", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "match\_only\_text", "count" : 51212, "index\_count" : 884, "script\_count" : 0 }, { "name" : "nested", "count" : 11652, "index\_count" : 926, "script\_count" : 0 }, { "name" : "object", "count" : 483247, "index\_count" : 1516, "script\_count" : 0 }, { "name" : "scaled\_float", "count" : 14302, "index\_count" : 1327, "script\_count" : 0 }, { "name" : "shape", "count" : 2, "index\_count" : 2, "script\_count" : 0 }, { "name" : "short", "count" : 3139, "index\_count" : 33, "script\_count" : 0 }, { "name" : "text", "count" : 37668, "index\_count" : 1643, "script\_count" : 0 }, { "name" : "version", "count" : 4, "index\_count" : 4, "script\_count" : 0 }, { "name" : "wildcard", "count" : 13263, "index\_count" : 887, "script\_count" : 0 } ], "runtime\_field\_types" : [ { "name" : "keyword", "count" : 3, "index\_count" : 3, "scriptless\_count" : 0, "shadowed\_count" : 0, "lang" : [ "painless" ], "lines\_max" : 1, "lines\_total" : 3, "chars\_max" : 31, "chars\_total" : 93, "source\_max" : 0, "source\_total" : 0, "doc\_max" : 1, "doc\_total" : 3 } ] }, "analysis" : { "char\_filter\_types" : , "tokenizer\_types" : , "filter\_types" : , "analyzer\_types" : , "built\_in\_char\_filters" : , "built\_in\_tokenizers" : , "built\_in\_filters" : , "built\_in\_analyzers" : [ { "name" : "simple", "count" : 6, "index\_count" : 3 } ] }, "versions" : [ { "version" : "7.10.0", "index\_count" : 11, "primary\_shard\_count" : 11, "total\_primary\_size" : "406.3mb", "total\_primary\_bytes" : 426052031 }, { "version" : "7.11.1", "index\_count" : 7, "primary\_shard\_count" : 9, "total\_primary\_size" : "24.1gb", "total\_primary\_bytes" : 25974561508 }, { "version" : "7.13.0", "index\_count" : 141, "primary\_shard\_count" : 171, "total\_primary\_size" : "57gb", "total\_primary\_bytes" : 61303241295 }, { "version" : "7.16.2", "index\_count" : 1514, "primary\_shard\_count" : 2140, "total\_primary\_size" : "487.7gb", "total\_primary\_bytes" : 523718585823 } ] }, "nodes" : { "count" : { "total" : 3, "coordinating\_only" : 0, "data" : 3, "data\_cold" : 3, "data\_content" : 3, "data\_frozen" : 3, "data\_hot" : 3, "data\_warm" : 3, "ingest" : 3, "master" : 3, "ml" : 3, "remote\_cluster\_client" : 3, "transform" : 3, "voting\_only" : 0 }, "versions" : [ "7.16.2" ], "os" : { "available\_processors" : 3, "allocated\_processors" : 3, "names" : [ { "name" : "Linux", "count" : 3 } ], "pretty\_names" : [ { "pretty\_name" : "Ubuntu 20.04.3 LTS", "count" : 3 } ], "architectures" : [ { "arch" : "amd64", "count" : 3 } ], "mem" : { "total" : "24gb", "total\_in\_bytes" : 25769803776, "free" : "617.3mb", "free\_in\_bytes" : 647376896, "used" : "23.3gb", "used\_in\_bytes" : 25122426880, "free\_percent" : 3, "used\_percent" : 97 } }, "process" : { "cpu" : { "percent" : 42 }, "open\_file\_descriptors" : { "min" : 7882, "max" : 8534, "avg" : 8142 } }, "jvm" : { "max\_uptime" : "2.4d", "max\_uptime\_in\_millis" : 210987876, "versions" : [ { "version" : "17.0.1", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "17.0.1+12", "vm\_vendor" : "Eclipse Adoptium", "bundled\_jdk" : true, "using\_bundled\_jdk" : true, "count" : 3 } ], "mem" : { "heap\_used" : "11.4gb", "heap\_used\_in\_bytes" : 12243000832, "heap\_max" : "18gb", "heap\_max\_in\_bytes" : 19327352832 }, "threads" : 505 }, "fs" : { "total" : "882.6gb", "total\_in\_bytes" : 947778945024, "free" : "305.7gb", "free\_in\_bytes" : 328301461504, "available" : "305.7gb", "available\_in\_bytes" : 328251129856 }, "plugins" : , "network\_types" : { "transport\_types" : { "security4" : 3 }, "http\_types" : { "security4" : 3 } }, "discovery\_types" : { "zen" : 3 }, "packaging\_types" : [ { "flavor" : "default", "type" : "docker", "count" : 3 } ], "ingest" : { "number\_of\_pipelines" : 45, "processor\_stats" : { "append" : { "count" : 99245, "failed" : 0, "current" : 0, "time" : "53ms", "time\_in\_millis" : 53 }, "conditional" : { "count" : 892430, "failed" : 0, "current" : 0, "time" : "7.6s", "time\_in\_millis" : 7617 }, "convert" : { "count" : 0, "failed" : 0, "current" : 0, "time" : "0s", "time\_in\_millis" : 0 }, "csv" : { "count" : 0, "failed" : 0, "current" : 0, "time" : "0s", "time\_in\_millis" : 0 }, "date" : { "count" : 194635, "failed" : 0, "current" : 0, "time" : "3.9s", "time\_in\_millis" : 3953 }, "geoip" : { "count" : 463665, "failed" : 0, "current" : 0, "time" : "13.5s", "time\_in\_millis" : 13519 }, "grok" : { "count" : 488970, "failed" : 455, "current" : 0, "time" : "6.8s", "time\_in\_millis" : 6833 }, "gsub" : { "count" : 0, "failed" : 0, "current" : 0, "time" : "0s", "time\_in\_millis" : 0 }, "pipeline" : { "count" : 564940, "failed" : 0, "current" : 0, "time" : "7s", "time\_in\_millis" : 7011 }, "remove" : { "count" : 665149, "failed" : 0, "current" : 0, "time" : "891ms", "time\_in\_millis" : 891 }, "rename" : { "count" : 649730, "failed" : 0, "current" : 0, "time" : "520ms", "time\_in\_millis" : 520 }, "script" : { "count" : 99245, "failed" : 0, "current" : 0, "time" : "2.4s", "time\_in\_millis" : 2435 }, "set" : { "count" : 379647, "failed" : 0, "current" : 0, "time" : "2.2s", "time\_in\_millis" : 2250 }, "set\_security\_user" : { "count" : 0, "failed" : 0, "current" : 0, "time" : "0s", "time\_in\_millis" : 0 }, "uppercase" : { "count" : 62, "failed" : 0, "current" : 0, "time" : "0s", "time\_in\_millis" : 0 }, "uri\_parts" : { "count" : 99245, "failed" : 5688, "current" : 0, "time" : "316ms", "time\_in\_millis" : 316 }, "urldecode" : { "count" : 96383, "failed" : 0, "current" : 0, "time" : "194ms", "time\_in\_millis" : 194 }, "user\_agent" : { "count" : 299588, "failed" : 61970, "current" : 0, "time" : "2.8s", "time\_in\_millis" : 2802 } } } } } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 7, 2022, 6:28am UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/4 "2022-11-07T06:28:04Z")
Please format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 > [@balaa](#): > > "\_nodes" : { > "total" : 3, > [@balaa](#): > > "shards" : { > "total" : 2487, This is probably why, you have far too many shards for your nodes and the amount of data in your cluster. You should look at how you can reduce that count. ---
**Author:** ![balaa](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@balaa](https://discuss.elastic.co/u/balaa)\ **Post date:** [November 7, 2022, 6:35am UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/5 "2022-11-07T06:35:09Z")
Hi Warkolm, You mean reduce the shards count ? i am getting this error only for this dashboard ? error msg like "Request error: node\_not\_connected\_exception, Node not connected & Request error: i\_o\_exception, Invalid string; unexpected character: 129 hex: 81" ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2022, 2:23pm UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/6 "2022-12-04T14:23:55Z")
---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 4, 2022, 9:06pm UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/7 "2022-12-04T21:06:29Z")
The issue is likely related to having too many shards for your cluster to efficiently manage, reducing the count reduces the load that is required to manage things. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 1, 2023, 4:08pm UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/8 "2023-01-01T16:08:04Z")
---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 29, 2023, 4:09pm UTC](https://discuss.elastic.co/t/shards-failure-kibana-dashboard/318287/9 "2023-01-29T16:09:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.