# Sharding Issue **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2018, 12:14pm UTC](https://discuss.elastic.co/t/sharding-issue/153593 "2018-10-23T12:14:16Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 12:14pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/1 "2018-10-23T12:14:16Z")
Team We are using the Dockerize ELK cluster and we are using logtrail pugin also in kibana.So we have 2 dashborad one is for discover and one is for logtrail But our shard limit is set as 1250 . So not we are facing issue your shard limit increase and our logtrail is not propely working . Could you please help me to fix this ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 23, 2018, 12:23pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/2 "2018-10-23T12:23:01Z")
How many shards do you have in the cluster? What is their average size? How many data nodes do you have in the cluster? What is their specification? ---
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 12:23pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/3 "2018-10-23T12:23:54Z")
> [@Christian\_Dahlqvist](#): > > hat is their average so we have define 1250 shard and data node 4 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 23, 2018, 12:27pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/4 "2018-10-23T12:27:33Z")
Which Elasticsearch version are you using? What is the specification of the nodes? How much heap do you have? Can you provide the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)? ---
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 12:29pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/5 "2018-10-23T12:29:32Z")
{ "\_nodes": { "total": 4, "successful": 4, "failed": 0 }, "cluster\_name": "ELK", "timestamp": 1540297696630, "status": "green", "indices": { "count": 814, "shards": { "total": 8132, "primaries": 4066, "replication": 1, "index": { "shards": { "min": 2, "max": 10, "avg": 9.99017199017199 }, "primaries": { "min": 1, "max": 5, "avg": 4.995085995085995 }, "replication": { "min": 1, "max": 1, "avg": 1 } } }, "docs": { "count": 616442429, "deleted": 0 }, "store": { "size": "291.4gb", "size\_in\_bytes": 312990804408, "throttle\_time": "0s", "throttle\_time\_in\_millis": 0 }, "fielddata": { "memory\_size": "0b", "memory\_size\_in\_bytes": 0, "evictions": 0 }, "query\_cache": { "memory\_size": "33.1mb", "memory\_size\_in\_bytes": 34745520, "total\_count": 3556933, "hit\_count": 3290542, "miss\_count": 266391, "cache\_size": 9208, "cache\_count": 9412, "evictions": 204 }, "completion": { "size": "0b", "size\_in\_bytes": 0 }, "segments": { "count": 41882, "memory": "1.6gb", "memory\_in\_bytes": 1754557450, "terms\_memory": "1.4gb", "terms\_memory\_in\_bytes": 1546693704, "stored\_fields\_memory": "127.2mb", "stored\_fields\_memory\_in\_bytes": 133459088, "term\_vectors\_memory": "0b", "term\_vectors\_memory\_in\_bytes": 0, "norms\_memory": "21.1mb", "norms\_memory\_in\_bytes": 22138560, "points\_memory": "13.7mb", "points\_memory\_in\_bytes": 14452842, "doc\_values\_memory": "36mb", "doc\_values\_memory\_in\_bytes": 37813256, "index\_writer\_memory": "0b", "index\_writer\_memory\_in\_bytes": 0, "version\_map\_memory": "0b", "version\_map\_memory\_in\_bytes": 0, "fixed\_bit\_set": "0b", "fixed\_bit\_set\_memory\_in\_bytes": 0, "max\_unsafe\_auto\_id\_timestamp": 1540252810171, "file\_sizes": {} } }, "nodes": { "count": { "total": 4, "data": 4, "coordinating\_only": 0, "master": 4, "ingest": 4 }, "versions": [ "5.3.0" ], "os": { "available\_processors": 32, "allocated\_processors": 32, "names": [ { "name": "Linux", "count": 4 } ], "mem": { "total": "125.6gb", "total\_in\_bytes": 134960480256, "free": "23.8gb", "free\_in\_bytes": 25566994432, "used": "101.8gb", "used\_in\_bytes": 109393485824, "free\_percent": 19, "used\_percent": 81 } }, "process": { "cpu": { "percent": 0 }, "open\_file\_descriptors": { "min": 4393, "max": 5077, "avg": 4895 } }, "jvm": { "max\_uptime": "84.5d", "max\_uptime\_in\_millis": 7308066849, "versions": [ { "version": "1.8.0\_121", "vm\_name": "OpenJDK 64-Bit Server VM", "vm\_version": "25.121-b13", "vm\_vendor": "Oracle Corporation", "count": 4 } ], "mem": { "heap\_used": "29.4gb", "heap\_used\_in\_bytes": 31670701560, "heap\_max": "47.9gb", "heap\_max\_in\_bytes": 51504480256 }, "threads": 414 }, "fs": { "total": "1tb", "total\_in\_bytes": 1162047315968, "free": "774.1gb", "free\_in\_bytes": 831253094400, "available": "725gb", "available\_in\_bytes": 778508849152, "spins": "true" }, "plugins": [], "network\_types": { "transport\_types": { "netty4": 4 }, "http\_types": { "netty4": 4 } } } } ---
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 12:30pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/6 "2018-10-23T12:30:21Z")
Node is very heavy but still we are getting issue not sure why ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 23, 2018, 12:34pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/7 "2018-10-23T12:34:53Z")
You have 8132 shards on 4 nodes with a total of just 48GB heap, which is far, far too much. Please read [this blog post for some practical guidance on shard sizes and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster), then rethink how you shard data and reduce the number of primary shards, consolidate indices and/or switch to e.g. weekly or monthly indices. You can also use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/indices-shrink-index.html) to reduce the number of shards for existing indices. ---
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 12:39pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/8 "2018-10-23T12:39:57Z")
But why we are getting this error **Error!** Exception while executing search query :[illegal\_argument\_exception] Trying to query 1600 shards, which is over the limit of 1200. This limit exists because querying many shards at the same time can make the job of the coordinating node very CPU and/or memory intensive. It is usually a better idea to have a smaller number of larger shards. Update [action.search.shard\_count.limit] to a greater value if you really want to query that many shards at the same time. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 23, 2018, 12:43pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/9 "2018-10-23T12:43:58Z")
There is a limit to the number of shards that can be queried in the version you are using. You are however severely oversharded and need to look at correcting that. ---
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 2:55pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/10 "2018-10-23T14:55:01Z")
What is the best way to fix this issue and correct the sharding ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 23, 2018, 2:55pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/11 "2018-10-23T14:55:44Z")
Did you look at the links I provided? ---
**Author:** ![abhinav.dwivedi](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@abhinav.dwivedi](https://discuss.elastic.co/u/abhinav.dwivedi)\ **Post date:** [October 23, 2018, 3:01pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/12 "2018-10-23T15:01:26Z")
Yes i have gone through the above link. Actually in our cluster we have more than one year data in ELK cluster . ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 23, 2018, 3:07pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/13 "2018-10-23T15:07:11Z")
I would recommend doing the following: - First change your sharding strategy going forward to prevent it from deteriorating. Consolidate indices with similar data, switch to monthly indices and reduce the number of primary shards to 1. - Then start reducing the number of shards already in the cluster. If you intend to keep data for a long time, consolidate indices and reindex into monthly indices with a single primary shard using the [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/docs-reindex.html). Just using the shrink index API will probably not bring you down far enough. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 20, 2018, 3:07pm UTC](https://discuss.elastic.co/t/sharding-issue/153593/14 "2018-11-20T15:07:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.