# Please explain the flow of data? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 21, 2014, 9:25pm UTC](https://discuss.elastic.co/t/please-explain-the-flow-of-data/16536 "2014-03-21T21:25:51Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [March 21, 2014, 9:25pm UTC](https://discuss.elastic.co/t/please-explain-the-flow-of-data/16536/1 "2014-03-21T21:25:51Z")
I'm trying to build a basic understanding of how indexing and searching works, hopefully someone can either point me to good resources or explain! I'm trying to figure out what having multiple "coordinator" nodes as defined in the elasticsearch.yml would do, and what having multiple "search load balancer" nodes would do. Both in the context of indexing and searching. Is there a functional difference between a "coordinator" node and a "search load balancer" node, beyond the fact that a "search load balancer" node can't be elected master? Say I have a 4 node cluster. There's a master only "coordinator" node, that doesn't store data, named "master". node.master: true node.data: false There are three data only nodes, "A", "B" and "C" node.master: false node.date: true I have an index "test" with two shards and one replica. Primary shard 0 lives on A, primary shard 1 lives on C, replica shard 0 lives on B, replica shard 1 lives on A. I send the command curl -XPOST [http://master:9200/test/test](http://master:9200/test/test) -d '{"foo":"bar"}' A connection is made to master, and the data is sent to master to be indexed. Master randomly decides to place this document in shard 1, so it gets sent to the primary shard 1 on C and replica shard 1 on B, right? This is where routing can come in, I can say that that document really should go to shard 0 because I said so. So this is a fairly simple scenario, assuming I'm correct. What benefit do I get to indexing when I add more "coordinator" nodes? node.master: true node.data: false What about if I add "search load balancer" nodes? node.master: false node.data: false How about on the searching side of things? I send a search to master, curl -XPOST [http://master:9200/test/test/\_search](http://master:9200/test/test/_search) -d '{"query":{"match\_all":{}}}' Master sends these queries off to A, B and C, who each generate their own results and return them to master. Each data node queries all the relevant shards that are present locally and then combines those results for delivery to master. Do only primary shards get queried, or are replica shards queried too? Master takes these combined results from all the relevant nodes and combines them into the final query response. Same questions: What benefit do I get to searching when I add more nodes that are like master? node.master: true node.data: false What about if I add "search load balancer" nodes? node.master: false node.data: false Is the only difference between a node.master: true node.data: false and a node.master: false node.data: false that the node is a candidate to be a master, should it be elected? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 21, 2014, 10:20pm UTC](https://discuss.elastic.co/t/please-explain-the-flow-of-data/16536/2 "2014-03-21T22:20:24Z")
A couple of things; 1. You should have n/2+1 masters in your cluster, where n = number of nodes. This helps prevent split brain situations and is best practise. 2. Your master nodes can store data, this way you don't need to add more nodes to fulfil the above. Your indexing scenario is correct. For searching, replica's and primaries can be queried. For both - Adding more masters adds redundancy as per the first two points. Adding more search nodes won't do much though other than reduce the load on your masters (unless someone else can add anything I don't know :p). And for your final question, yes that is correct. To give you an idea of practical application, we don't use search nodes but have 3 non-data masters that handle all queries, and a bunch of data only nodes for storing everything. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 22 March 2014 08:25, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > I'm trying to build a basic understanding of how indexing and searching > works, hopefully someone can either point me to good resources or explain! > I'm trying to figure out what having multiple "coordinator" nodes as > defined in the elasticsearch.yml would do, and what having multiple "search > load balancer" nodes would do. Both in the context of indexing and > searching. > Is there a functional difference between a "coordinator" node and a > "search load balancer" node, beyond the fact that a "search load balancer" > node can't be elected master? > > Say I have a 4 node cluster. There's a master only "coordinator" node, > that doesn't store data, named "master". > node.master: true > node.data: false > > There are three data only nodes, "A", "B" and "C" > node.master: false > node.date: true > > I have an index "test" with two shards and one replica. Primary shard 0 > lives on A, primary shard 1 lives on C, replica shard 0 lives on B, replica > shard 1 lives on A. > > I send the command > curl -XPOST [http://master:9200/test/test](http://master:9200/test/test) -d '{"foo":"bar"}' > > A connection is made to master, and the data is sent to master to be > indexed. Master randomly decides to place this document in shard 1, so it > gets sent to the primary shard 1 on C and replica shard 1 on B, right? This > is where routing can come in, I can say that that document really should go > to shard 0 because I said so. > > So this is a fairly simple scenario, assuming I'm correct. > > What benefit do I get to indexing when I add more "coordinator" nodes? > node.master: true > node.data: false > > What about if I add "search load balancer" nodes? > node.master: false > node.data: false > > How about on the searching side of things? > I send a search to master, > curl -XPOST [http://master:9200/test/test/\_search](http://master:9200/test/test/_search) -d > '{"query":{"match\_all":{}}}' > > Master sends these queries off to A, B and C, who each generate their own > results and return them to master. Each data node queries all the relevant > shards that are present locally and then combines those results for > delivery to master. Do only primary shards get queried, or are replica > shards queried too? > Master takes these combined results from all the relevant nodes and > combines them into the final query response. > > Same questions: > What benefit do I get to searching when I add more nodes that are like > master? > node.master: true > node.data: false > > What about if I add "search load balancer" nodes? > node.master: false > node.data: false > > Is the only difference between a > node.master: true > node.data: false > and a > node.master: false > node.data: false > that the node is a candidate to be a master, should it be elected? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YebkNK-nJgH63qP2p0pbw4ctUxVoArHYvT0qXDXmPsbQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YebkNK-nJgH63qP2p0pbw4ctUxVoArHYvT0qXDXmPsbQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [March 21, 2014, 10:36pm UTC](https://discuss.elastic.co/t/please-explain-the-flow-of-data/16536/3 "2014-03-21T22:36:43Z")
Awesome, ok, thank you. Is the logic behind not allowing storage on master nodes to both: Take advantage of a system with limited storage resources and Have a dedicated results aggregator/search handler? I can imagine if I had a particularly badly written gnarly search, trying to deal with the results on a master and a querying the results at the same time could be bad. So in a 16 node cluster you'd want to have 9 nodes allowed to be masters, (n/2)+1? Thanks again! Josh On Friday, March 21, 2014 3:20:24 PM UTC-7, Mark Walkom wrote: > A couple of things; > > 1. You should have n/2+1 masters in your cluster, where n = number of > nodes. This helps prevent split brain situations and is best practise. > 2. Your master nodes can store data, this way you don't need to add > more nodes to fulfil the above. > > Your indexing scenario is correct. > For searching, replica's and primaries can be queried. > For both - Adding more masters adds redundancy as per the first two > points. Adding more search nodes won't do much though other than reduce the > load on your masters (unless someone else can add anything I don't know :p). > > And for your final question, yes that is correct. > > To give you an idea of practical application, we don't use search nodes > but have 3 non-data masters that handle all queries, and a bunch of data > only nodes for storing everything. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 22 March 2014 08:25, Josh Harrison \<[hij...@gmail.com](mailto:hij...@gmail.com) \\>wrote: > > > I'm trying to build a basic understanding of how indexing and searching > > works, hopefully someone can either point me to good resources or explain! > > I'm trying to figure out what having multiple "coordinator" nodes as > > defined in the elasticsearch.yml would do, and what having multiple "search > > load balancer" nodes would do. Both in the context of indexing and > > searching. > > Is there a functional difference between a "coordinator" node and a > > "search load balancer" node, beyond the fact that a "search load balancer" > > node can't be elected master? > > > > Say I have a 4 node cluster. There's a master only "coordinator" node, > > that doesn't store data, named "master". > > node.master: true > > node.data: false > > > > There are three data only nodes, "A", "B" and "C" > > node.master: false > > node.date: true > > > > I have an index "test" with two shards and one replica. Primary shard 0 > > lives on A, primary shard 1 lives on C, replica shard 0 lives on B, replica > > shard 1 lives on A. > > > > I send the command > > curl -XPOST [http://master:9200/test/test](http://master:9200/test/test) -d '{"foo":"bar"}' > > > > A connection is made to master, and the data is sent to master to be > > indexed. Master randomly decides to place this document in shard 1, so it > > gets sent to the primary shard 1 on C and replica shard 1 on B, right? This > > is where routing can come in, I can say that that document really should go > > to shard 0 because I said so. > > > > So this is a fairly simple scenario, assuming I'm correct. > > > > What benefit do I get to indexing when I add more "coordinator" nodes? > > node.master: true > > node.data: false > > > > What about if I add "search load balancer" nodes? > > node.master: false > > node.data: false > > > > How about on the searching side of things? > > I send a search to master, > > curl -XPOST [http://master:9200/test/test/\_search](http://master:9200/test/test/_search) -d > > '{"query":{"match\_all":{}}}' > > > > Master sends these queries off to A, B and C, who each generate their own > > results and return them to master. Each data node queries all the relevant > > shards that are present locally and then combines those results for > > delivery to master. Do only primary shards get queried, or are replica > > shards queried too? > > Master takes these combined results from all the relevant nodes and > > combines them into the final query response. > > > > Same questions: > > What benefit do I get to searching when I add more nodes that are like > > master? > > node.master: true > > node.data: false > > > > What about if I add "search load balancer" nodes? > > node.master: false > > node.data: false > > > > Is the only difference between a > > node.master: true > > node.data: false > > and a > > node.master: false > > node.data: false > > that the node is a candidate to be a master, should it be elected? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5b45303b-b012-4c3c-9bd7-86cf02d7f937%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5b45303b-b012-4c3c-9bd7-86cf02d7f937%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 21, 2014, 10:50pm UTC](https://discuss.elastic.co/t/please-explain-the-flow-of-data/16536/4 "2014-03-21T22:50:40Z")
Yes you can leverage a master to be a search node in that way. We have a 15 node cluster with 3 masters, I'm thinking I'll add another 2 when we add a few more data nodes in the next few weeks. Essentially you want an uneven number of masters to ensure a quorum is reached. But when you start getting large clusters, ie tens of nodes, it doesn't make as much sense to have n/2+1 masters. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 22 March 2014 09:36, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > Awesome, ok, thank you. > Is the logic behind not allowing storage on master nodes to both: > Take advantage of a system with limited storage resources > and > Have a dedicated results aggregator/search handler? > > I can imagine if I had a particularly badly written gnarly search, trying > to deal with the results on a master and a querying the results at the same > time could be bad. > > So in a 16 node cluster you'd want to have 9 nodes allowed to be masters, > (n/2)+1? > > Thanks again! > Josh > > On Friday, March 21, 2014 3:20:24 PM UTC-7, Mark Walkom wrote: > > > A couple of things; > > > > 1. You should have n/2+1 masters in your cluster, where n = number of > > nodes. This helps prevent split brain situations and is best practise. > > 2. Your master nodes can store data, this way you don't need to add > > more nodes to fulfil the above. > > > > Your indexing scenario is correct. > > For searching, replica's and primaries can be queried. > > For both - Adding more masters adds redundancy as per the first two > > points. Adding more search nodes won't do much though other than reduce the > > load on your masters (unless someone else can add anything I don't know :p). > > > > And for your final question, yes that is correct. > > > > To give you an idea of practical application, we don't use search nodes > > but have 3 non-data masters that handle all queries, and a bunch of data > > only nodes for storing everything. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 22 March 2014 08:25, Josh Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > > > > > I'm trying to build a basic understanding of how indexing and searching > > > works, hopefully someone can either point me to good resources or explain! > > > I'm trying to figure out what having multiple "coordinator" nodes as > > > defined in the elasticsearch.yml would do, and what having multiple "search > > > load balancer" nodes would do. Both in the context of indexing and > > > searching. > > > Is there a functional difference between a "coordinator" node and a > > > "search load balancer" node, beyond the fact that a "search load balancer" > > > node can't be elected master? > > > > > > Say I have a 4 node cluster. There's a master only "coordinator" node, > > > that doesn't store data, named "master". > > > node.master: true > > > node.data: false > > > > > > There are three data only nodes, "A", "B" and "C" > > > node.master: false > > > node.date: true > > > > > > I have an index "test" with two shards and one replica. Primary shard 0 > > > lives on A, primary shard 1 lives on C, replica shard 0 lives on B, replica > > > shard 1 lives on A. > > > > > > I send the command > > > curl -XPOST [http://master:9200/test/test](http://master:9200/test/test) -d '{"foo":"bar"}' > > > > > > A connection is made to master, and the data is sent to master to be > > > indexed. Master randomly decides to place this document in shard 1, so it > > > gets sent to the primary shard 1 on C and replica shard 1 on B, right? This > > > is where routing can come in, I can say that that document really should go > > > to shard 0 because I said so. > > > > > > So this is a fairly simple scenario, assuming I'm correct. > > > > > > What benefit do I get to indexing when I add more "coordinator" nodes? > > > node.master: true > > > node.data: false > > > > > > What about if I add "search load balancer" nodes? > > > node.master: false > > > node.data: false > > > > > > How about on the searching side of things? > > > I send a search to master, > > > curl -XPOST [http://master:9200/test/test/\_search](http://master:9200/test/test/_search) -d > > > '{"query":{"match\_all":{}}}' > > > > > > Master sends these queries off to A, B and C, who each generate their > > > own results and return them to master. Each data node queries all the > > > relevant shards that are present locally and then combines those results > > > for delivery to master. Do only primary shards get queried, or are replica > > > shards queried too? > > > Master takes these combined results from all the relevant nodes and > > > combines them into the final query response. > > > > > > Same questions: > > > What benefit do I get to searching when I add more nodes that are like > > > master? > > > node.master: true > > > node.data: false > > > > > > What about if I add "search load balancer" nodes? > > > node.master: false > > > node.data: false > > > > > > Is the only difference between a > > > node.master: true > > > node.data: false > > > and a > > > node.master: false > > > node.data: false > > > that the node is a candidate to be a master, should it be elected? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9% > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eaff1d85-1e85-422d-bfba-9a0825ed5da9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5b45303b-b012-4c3c-9bd7-86cf02d7f937%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5b45303b-b012-4c3c-9bd7-86cf02d7f937%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5b45303b-b012-4c3c-9bd7-86cf02d7f937%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5b45303b-b012-4c3c-9bd7-86cf02d7f937%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YaRJPq\_T8GKDuVyZjjmFHE0JZQ36Vo-8GrTk0JOTNpvg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YaRJPq_T8GKDuVyZjjmFHE0JZQ36Vo-8GrTk0JOTNpvg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:41am UTC](https://discuss.elastic.co/t/please-explain-the-flow-of-data/16536/5 "2017-07-06T01:41:28Z")