# Logstash-Elasticsearch looking for advise on cluster and shard/index routing **URL:** \ **Category:** Elasticsearch\ **Created:** [February 17, 2014, 3:32pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-looking-for-advise-on-cluster-and-shard-index-routing/15850 "2014-02-17T15:32:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Fernando\_Emwferm](https://avatars.discourse-cdn.com/v4/letter/f/898d66/32.png) [@Fernando\_Emwferm](https://discuss.elastic.co/u/Fernando_Emwferm)\ **Post date:** [February 17, 2014, 3:32pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-looking-for-advise-on-cluster-and-shard-index-routing/15850/1 "2014-02-17T15:32:35Z")
Hej to y'all: I am looking for some guidance regarding elasticsearch together with logstash. I am new to all of these and I want to find if the setup we want is possible. We are setting up a couple of nodes in AWS EC2 which are located in different availability zones within AWS. I have resolved the issue of setting up a Cluster inside AWS without the plugin just using security groups and iptables. But the setup that we want requires the cluster mainly for searching. A bigger issue for us is about cost since AWS charges for the data transfering between availability zones, and that, we want to avoid. So back to what our scenario looks like, we would like to have the following: We want to have two separate servers that have all their indexes and shards located locally (meaning that when indexing information via logstash we need that all new and existing shards are not spread across the two nodes in the cluster). Node A will receive log information from availability zone 1 and needs to keep that info only in that node. And Node B will receive log information from availability zone 2 with the same behavior. But (and here is where I have an issue with) we would like to be able to go into one server (lets say Node A) and be able to query information from both Node A and Node B. Our setup is not critical and that is why we can skip the nice clustering and distributed functionality of elasticsearch. I have tried (after a lot of reading and "googling") with the parameters cluster.routing.allocation.awareness.attributes, cluster.routing.allocation.awareness.force.zone.values, node.zone, index.routing.allocation.total\_shards\_per\_node, index.routing.allocation.require.zone and index.routing.allocation.exclude.zone. I have set these in the elasticsearch.yml file starting the elasticsearch cluster and I also put these values in elasticsearch.yml file for logstash. But with every try (I delete the data directory everytime to test) when I index some test logs from Node B some shards are written in Node A anyway. Do you think this setup is at all possible??? or maybe the elasticsearch distributed behavior cannot be changed like this and that is ok as long as I know then I can move on with another setup since, as I said, is nothing critical but I cannot take much more time investigating this (already been at it for a couple of weeks). Thank you to anyone for their time, attention and help, best regards, s.r./Fernando P.S.: I am using logstash 1.3.3 and elasticsearch 0.90.9 with Kibana 3 Milestone 4. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a770c8dd-6498-4722-9e6e-4fd034523434%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a770c8dd-6498-4722-9e6e-4fd034523434%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\ **Post date:** [February 17, 2014, 5:36pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-looking-for-advise-on-cluster-and-shard-index-routing/15850/2 "2014-02-17T17:36:30Z")
Hi, I just read online an article which although doesn't describe your scenario exactly, might be helpful [http://blog.qbox.io/launching-and-scaling-elasticsearch](http://blog.qbox.io/launching-and-scaling-elasticsearch) My guess is that you're simply setting up two boxes with the following configuration index.number\_of\_shards: 1 index.number\_of\_replicas: 0 I don't know how to control data staying on the Server you input data, maybe someone can comment here. With multiple shards in my testing, I have not found any way to control where the shard is allocated. Maybe disable load balancing? I would expect that any query you run against your cluster will be executed on both nodes. Tony On Monday, February 17, 2014 7:32:35 AM UTC-8, Fernando Emwferm wrote: > Hej to y'all: > > I am looking for some guidance regarding elasticsearch together with > logstash. I am new to all of these and I want to find if the setup we want > is possible. > > We are setting up a couple of nodes in AWS EC2 which are located in > different availability zones within AWS. I have resolved the issue of > setting up a Cluster inside AWS without the plugin just using security > groups and iptables. But the setup that we want requires the cluster mainly > for searching. A bigger issue for us is about cost since AWS charges for > the data transfering between availability zones, and that, we want to avoid. > > So back to what our scenario looks like, we would like to have the > following: > > We want to have two separate servers that have all their indexes and > shards located locally (meaning that when indexing information via logstash > we need that all new and existing shards are not spread across the two > nodes in the cluster). Node A will receive log information from > availability zone 1 and needs to keep that info only in that node. And Node > B will receive log information from availability zone 2 with the same > behavior. But (and here is where I have an issue with) we would like to be > able to go into one server (lets say Node A) and be able to query > information from both Node A and Node B. Our setup is not critical and that > is why we can skip the nice clustering and distributed functionality of > elasticsearch. > > I have tried (after a lot of reading and "googling") with the parameters > cluster.routing.allocation.awareness.attributes, > cluster.routing.allocation.awareness.force.zone.values, node.zone, > index.routing.allocation.total\_shards\_per\_node, > index.routing.allocation.require.zone and > index.routing.allocation.exclude.zone. I have set these in the > elasticsearch.yml file starting the elasticsearch cluster and I also put > these values in elasticsearch.yml file for logstash. But with every try (I > delete the data directory everytime to test) when I index some test logs > from Node B some shards are written in Node A anyway. > > Do you think this setup is at all possible??? or maybe the elasticsearch > distributed behavior cannot be changed like this and that is ok as long as > I know then I can move on with another setup since, as I said, is nothing > critical but I cannot take much more time investigating this (already been > at it for a couple of weeks). > > Thank you to anyone for their time, attention and help, best regards, > s.r./Fernando > P.S.: I am using logstash 1.3.3 and elasticsearch 0.90.9 with Kibana 3 > Milestone 4. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b2d9daab-8af6-4f2e-9da6-af93891798ab%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2d9daab-8af6-4f2e-9da6-af93891798ab%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 18, 2014, 10:49pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-looking-for-advise-on-cluster-and-shard-index-routing/15850/3 "2014-02-18T22:49:42Z")
What you probably really want is to eventually migrate to 1.0 and then build 2 separate clusters (one in each AZ) and then index into them separately, and then use the tribe node ([http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html)) to federate your search results across clusters. In the meantime assuming you really want a single cluster, there's no reason why you can't use index allocation awareness to do what you want. So for example: ES.yml on zone 1 would have: node.zone: zone1 ES.yml on zone 2 would have: node.zone: zone2 Then when you create an index that you want only to go to zone 1, you will say something like (maybe just use an index template for convenience): PUT [http://localhost:9200/](http://localhost:9200/) { "settings": { "index.routing.allocation.include.zone": "zone1" } } And when you create an index that you want only to go to zone 2, you will say something like (maybe just use an index template for convenience): PUT [http://localhost:9200/](http://localhost:9200/) { "settings": { "index.routing.allocation.include.zone": "zone2" } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b8444c3-dfa3-464c-b974-ada0b09be62b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b8444c3-dfa3-464c-b974-ada0b09be62b%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:49am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-looking-for-advise-on-cluster-and-shard-index-routing/15850/4 "2017-07-06T01:49:15Z")