# Cluster misconfigured or elastic is just SLOW!? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 22, 2015, 6:45pm UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784 "2015-01-22T18:45:23Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Sam\_Flint](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sam_flint/32/967_2.png) [@Sam\_Flint](https://discuss.elastic.co/u/Sam_Flint)\ **Post date:** [January 22, 2015, 6:45pm UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784/1 "2015-01-22T18:45:23Z")
Hi, I am evaluating elastic search for a data warehouse project. I have a 9 node cluster with 1 replica. I have loaded 6 days worth of data. Things seem sluggish all around. Here is the health of the nodes { "active\_primary\_shards": 9251, "active\_shards": 13509, "cluster\_name": "qa-elasticsearch", "initializing\_shards": 23, "number\_of\_data\_nodes": 9, "number\_of\_nodes": 9, "relocating\_shards": 0, "status": "red", "timed\_out": false, "unassigned\_shards": 5054 } I am wondering if I am missing some configuration that will speed things up or if I did something wrong? I see the status is "RED" and it seems that there are still node initializing and the cluster has been up for over 24 hours. I used logstash to push the data into elastic. Any help would be appreciated....or if any additional information is needed I can provide. Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Kimbro\_Staken](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Kimbro\_Staken](https://discuss.elastic.co/u/Kimbro_Staken)\ **Post date:** [January 22, 2015, 7:04pm UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784/2 "2015-01-22T19:04:21Z")
Yes you have something very wrong. That is showing you have a huge number of shards and the cluster is obviously struggling to allocate all of them. You said you have 9 nodes and 1 replica but you didn't specify how many shards per index? Kimbro On Thu, Jan 22, 2015 at 11:45 AM, Sam Flint [sam.flint@magnetic.com](mailto:sam.flint@magnetic.com) wrote: > Hi, > I am evaluating Elasticsearch for a data warehouse project. I have a > 9 node cluster with 1 replica. I have loaded 6 days worth of data. > Things seem sluggish all around. Here is the health of the nodes > > { > "active\_primary\_shards": 9251, > "active\_shards": 13509, > "cluster\_name": "qa-elasticsearch", > "initializing\_shards": 23, > "number\_of\_data\_nodes": 9, > "number\_of\_nodes": 9, > "relocating\_shards": 0, > "status": "red", > "timed\_out": false, > "unassigned\_shards": 5054 > } > > I am wondering if I am missing some configuration that will speed things > up or if I did something wrong? I see the status is "RED" and it seems > that there are still node initializing and the cluster has been up for over > 24 hours. I used logstash to push the data into elastic. > Any help would be appreciated....or if any additional information is > needed I can provide. > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbpxtS81ogsy2ZjQ1sabRoWa4B%2BLUMt7FerPR2Y70FWNg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbpxtS81ogsy2ZjQ1sabRoWa4B%2BLUMt7FerPR2Y70FWNg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 22, 2015, 8:32pm UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784/3 "2015-01-22T20:32:07Z")
That's a massive number of shards! As Kimbro pointed out, that's likely your problem. On 23 January 2015 at 06:04, Kimbro Staken [kstaken@kstaken.com](mailto:kstaken@kstaken.com) wrote: > Yes you have something very wrong. That is showing you have a huge number > of shards and the cluster is obviously struggling to allocate all of them. > You said you have 9 nodes and 1 replica but you didn't specify how many > shards per index? > > Kimbro > > On Thu, Jan 22, 2015 at 11:45 AM, Sam Flint [sam.flint@magnetic.com](mailto:sam.flint@magnetic.com) > wrote: > > > Hi, > > I am evaluating Elasticsearch for a data warehouse project. I have > > a 9 node cluster with 1 replica. I have loaded 6 days worth of data. > > Things seem sluggish all around. Here is the health of the nodes > > > > { > > "active\_primary\_shards": 9251, > > "active\_shards": 13509, > > "cluster\_name": "qa-elasticsearch", > > "initializing\_shards": 23, > > "number\_of\_data\_nodes": 9, > > "number\_of\_nodes": 9, > > "relocating\_shards": 0, > > "status": "red", > > "timed\_out": false, > > "unassigned\_shards": 5054 > > } > > > > I am wondering if I am missing some configuration that will speed things > > up or if I did something wrong? I see the status is "RED" and it seems > > that there are still node initializing and the cluster has been up for over > > 24 hours. I used logstash to push the data into elastic. > > Any help would be appreciated....or if any additional information is > > needed I can provide. > > > > Thanks > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a1c4cb64-82f5-4ab1-be6a-9b29e0a01be9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbpxtS81ogsy2ZjQ1sabRoWa4B%2BLUMt7FerPR2Y70FWNg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbpxtS81ogsy2ZjQ1sabRoWa4B%2BLUMt7FerPR2Y70FWNg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbpxtS81ogsy2ZjQ1sabRoWa4B%2BLUMt7FerPR2Y70FWNg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbpxtS81ogsy2ZjQ1sabRoWa4B%2BLUMt7FerPR2Y70FWNg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_PVuWo4Y5J%2BcVr4Fy9f8iQnvd8xtt-%3D9v10Q-wnVVYEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_PVuWo4Y5J%2BcVr4Fy9f8iQnvd8xtt-%3D9v10Q-wnVVYEg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sam\_Flint](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sam_flint/32/967_2.png) [@Sam\_Flint](https://discuss.elastic.co/u/Sam_Flint)\ **Post date:** [January 23, 2015, 3:28pm UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784/4 "2015-01-23T15:28:40Z")
Here is the .yml part for the shards # Set the number of shards (splits) of an index (5 by default): # index.number\_of\_shards: 64 # Set the number of replicas (additional copies) of an index (1 by default): # index.number\_of\_replicas: 1 I have it set to 64 shards and 1 replica. Is there a recommended amount of shards that I should be using? On Thursday, January 22, 2015 at 1:45:23 PM UTC-5, Sam Flint wrote: > Hi, > I am evaluating Elasticsearch for a data warehouse project. I have a > 9 node cluster with 1 replica. I have loaded 6 days worth of data. > Things seem sluggish all around. Here is the health of the nodes > > { > "active\_primary\_shards": 9251, > "active\_shards": 13509, > "cluster\_name": "qa-elasticsearch", > "initializing\_shards": 23, > "number\_of\_data\_nodes": 9, > "number\_of\_nodes": 9, > "relocating\_shards": 0, > "status": "red", > "timed\_out": false, > "unassigned\_shards": 5054 > } > > I am wondering if I am missing some configuration that will speed things > up or if I did something wrong? I see the status is "RED" and it seems > that there are still node initializing and the cluster has been up for over > 24 hours. I used logstash to push the data into elastic. > Any help would be appreciated....or if any additional information is > needed I can provide. > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/244fbf99-5ec4-4052-a860-16fad53159ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/244fbf99-5ec4-4052-a860-16fad53159ed%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Kimbro\_Staken](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Kimbro\_Staken](https://discuss.elastic.co/u/Kimbro_Staken)\ **Post date:** [January 23, 2015, 4:49pm UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784/5 "2015-01-23T16:49:28Z")
How many shards to use is a complicated question and depends on the specific use case. For testing in this scenario though, it's likely that just matching the number of nodes you have would be a good choice. Then you will have 1 primary shard for each index on each node. That said it also looks like maybe you're creating more indices than is ideal as well. 64 shards with 1 replica and 6 days data in daily indices should only produce 768 total shards. Looks to me like maybe you have 1 index per hour? On Fri, Jan 23, 2015 at 8:28 AM, Sam Flint [sam.flint@magnetic.com](mailto:sam.flint@magnetic.com) wrote: > Here is the .yml part for the shards > > # Set the number of shards (splits) of an index (5 by default): > > # > > index.number\_of\_shards: 64 > > # Set the number of replicas (additional copies) of an index (1 by > > default): > > # > > index.number\_of\_replicas: 1 > > I have it set to 64 shards and 1 replica. Is there a recommended amount > of shards that I should be using? > > On Thursday, January 22, 2015 at 1:45:23 PM UTC-5, Sam Flint wrote: > > > Hi, > > I am evaluating Elasticsearch for a data warehouse project. I have > > a 9 node cluster with 1 replica. I have loaded 6 days worth of data. > > Things seem sluggish all around. Here is the health of the nodes > > > > { > > "active\_primary\_shards": 9251, > > "active\_shards": 13509, > > "cluster\_name": "qa-elasticsearch", > > "initializing\_shards": 23, > > "number\_of\_data\_nodes": 9, > > "number\_of\_nodes": 9, > > "relocating\_shards": 0, > > "status": "red", > > "timed\_out": false, > > "unassigned\_shards": 5054 > > } > > > > I am wondering if I am missing some configuration that will speed things > > up or if I did something wrong? I see the status is "RED" and it seems > > that there are still node initializing and the cluster has been up for over > > 24 hours. I used logstash to push the data into elastic. > > Any help would be appreciated....or if any additional information is > > needed I can provide. > > > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/244fbf99-5ec4-4052-a860-16fad53159ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/244fbf99-5ec4-4052-a860-16fad53159ed%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/244fbf99-5ec4-4052-a860-16fad53159ed%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/244fbf99-5ec4-4052-a860-16fad53159ed%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbZRa-\_05-eKuF7Tej88q4u1gtK4RB9b2ypTDn1gfacBw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAA0DmXbZRa-_05-eKuF7Tej88q4u1gtK4RB9b2ypTDn1gfacBw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:37am UTC](https://discuss.elastic.co/t/cluster-misconfigured-or-elastic-is-just-slow/21784/6 "2017-07-06T00:37:01Z")