# Configuration Brain Wobbles **URL:** \ **Category:** Elasticsearch\ **Created:** [July 30, 2014, 10:31pm UTC](https://discuss.elastic.co/t/configuration-brain-wobbles/18989 "2014-07-30T22:31:53Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Christopher\_Ambler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christopher_ambler/32/1353_2.png) [@Christopher\_Ambler](https://discuss.elastic.co/u/Christopher_Ambler)\ **Post date:** [July 30, 2014, 10:31pm UTC](https://discuss.elastic.co/t/configuration-brain-wobbles/18989/1 "2014-07-30T22:31:53Z")
I have a cluster with six nodes. The nodes are in different data centers, but I don't think that matters, as the connectivity is beefy and thick. I have turned multicast off and unicast on. Each node knows about all the others explicitly. When I bring up a visualization of the cluster using the "head" plugin, I see them all. This appears to work as it should. My cluster looks like this: DEV-02 (development data center) MESA-01 (mesa data center) MESA-02 (mesa data center) MESA-03 (mesa data center) BUCK-01 (buck data center) BUCK-02 (buck data center) I have each node configured for 5 shards. I have each node set to be master true and data true I do ALL of my document addition using MESA-01 and I can do queries on ANY node and get a result, so that's working. But I notice two things and have one requirement I can't figure out: 1. Most queries come in sub-30ms. But every now and again I get a query that is longer. I set my slow query log to complain over 100ms and I see that maybe one query out of 15 or so takes 800ms to 1200ms. This is on any node. 2. I have unassigned shards. I presume this is bad, yes? How do I get them to allocate? When I stop and start the service on any of the nodes, the shards are shuffled around, but rarely are the unassigned shards put on a node. Why? How do I resolve this? And my requirement - Is there a way to say, "Look, Elasticsearch, I don't want you shuffling shards around here and there, I'd like EVERY node to have a COMPLETE replica of the data, and you just keep it up to date. That way, you see, a query on a buck data center node won't have to ask a mesa data center for a document if it doesn't have it." Solving #2 is important, but solving my requirement is somewhat critical. I think fixing these two things will take care of issue #1. At least it'll get me configured right so if #1 is still there, I can diagnose from a position of not wondering if misconfiguration is my problem. Help? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 30, 2014, 11:01pm UTC](https://discuss.elastic.co/t/configuration-brain-wobbles/18989/2 "2014-07-30T23:01:40Z")
Standard response to this is ES is not built for multi DC clustering, but as long as you are aware you are of that then it's fine. Have you looked at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. ? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 31 July 2014 08:31, Christopher Ambler [chris@insiderhouse.com](mailto:chris@insiderhouse.com) wrote: > I have a cluster with six nodes. The nodes are in different data centers, > but I don't think that matters, as the connectivity is beefy and thick. I > have turned multicast off and unicast on. Each node knows about all the > others explicitly. When I bring up a visualization of the cluster using > the "head" plugin, I see them all. This appears to work as it should. My > cluster looks like this: > > DEV-02 (development data center) > MESA-01 (mesa data center) > MESA-02 (mesa data center) > MESA-03 (mesa data center) > BUCK-01 (buck data center) > BUCK-02 (buck data center) > > I have each node configured for 5 shards. > I have each node set to be master true and data true > > I do ALL of my document addition using MESA-01 and I can do queries on ANY > node and get a result, so that's working. But I notice two things and have > one requirement I can't figure out: > > 1. Most queries come in sub-30ms. But every now and again I get a query > that is longer. I set my slow query log to complain over 100ms and I see > that maybe one query out of 15 or so takes 800ms to 1200ms. This is on any > node. > > 2. I have unassigned shards. I presume this is bad, yes? How do I get them > to allocate? When I stop and start the service on any of the nodes, the > shards are shuffled around, but rarely are the unassigned shards put on a > node. Why? How do I resolve this? > > And my requirement - Is there a way to say, "Look, Elasticsearch, I don't > want you shuffling shards around here and there, I'd like EVERY node to > have a COMPLETE replica of the data, and you just keep it up to date. That > way, you see, a query on a buck data center node won't have to ask a mesa > data center for a document if it doesn't have it." > > Solving #2 is important, but solving my requirement is somewhat critical. > I think fixing these two things will take care of issue #1. > > At least it'll get me configured right so if #1 is still there, I can > diagnose from a position of not wondering if misconfiguration is my problem. > > Help? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZRF-uK7-dhuOSVqNwyQ13t6E3fGWkc4PpjvP\_e2F9Z4A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZRF-uK7-dhuOSVqNwyQ13t6E3fGWkc4PpjvP_e2F9Z4A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Christopher\_Ambler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christopher_ambler/32/1353_2.png) [@Christopher\_Ambler](https://discuss.elastic.co/u/Christopher_Ambler)\ **Post date:** [July 30, 2014, 11:53pm UTC](https://discuss.elastic.co/t/configuration-brain-wobbles/18989/3 "2014-07-30T23:53:04Z")
Yes, I'm aware of the multi-DC issue 😉 So yeah, this morning I dove into index shard allocation, and did just that. 3 zones (using node.zone as my tag) and then I set things to have 1 primary and 2 replications and then set it such that each zone should play along with this. This worked as advertised. One zone gets the primary and the other two zones get the two replications. Each zone as 2 nodes and I have 5 shards, so one node gets two and the other gets three. Head shows me this and it all makes sense. We're now doing about 30 searches per second, and I'm still seeing, about every 5 or 6 seconds, a single "slow query" in the 600ms to 900ms range. All other queries are sub-50ms. I need to find out why I'm seeing these consistent 600ms+ queries and eliminate them if I can. Everything else looks good. Oh, and on the unallocated shards issue, I tracked that down to having had more replications when I built the index and then taking my replications down. I had shards that had nowhere to go. So I just removed those indexes (they were old) and everything is green. So the consistent 600ms slow queries is my only issue now. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/99286c26-9816-4174-8358-a44438b448aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/99286c26-9816-4174-8358-a44438b448aa%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:12am UTC](https://discuss.elastic.co/t/configuration-brain-wobbles/18989/4 "2017-07-06T01:12:05Z")
Closed on July 6, 2017, 1:12am UTC