# Flume, Amazon and the Master node

**URL:** <https://discuss.elastic.co/t/flume-amazon-and-the-master-node/14197>\
**Category:** Elasticsearch\
**Created:** [October 31, 2013, 5:18pm UTC](https://discuss.elastic.co/t/flume-amazon-and-the-master-node/14197 "2013-10-31T17:18:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dipesh\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dipesh_patel/32/1579_2.png) [@Dipesh\_Patel](https://discuss.elastic.co/u/Dipesh_Patel)\
**Post date:** [October 31, 2013, 5:18pm UTC](https://discuss.elastic.co/t/flume-amazon-and-the-master-node/14197/1 "2013-10-31T17:18:24Z")

</div>

Hello fellow elasticsearch comrades

I've recently started looking at elasticsearch. Specifically deploying it  
to AWS. I have a few question about the clustering set up which I hope you  
can help with. It mostly stems from a bit of confussion about what the  
'master' node does.

We are currently using the flume elasticsearch sink to push log data into  
our elasticsearch cluster. The cluster sits behind a load balancer, hence  
flume is configured with a cname in it and not a list of master nodes.  
We've noticed that when we redeploy our cluster, flume looses network  
connection to the cluster and this doesn't fix itself when the cluster  
comes back. We have to restart flume then everthing works fine.

We traced this to the fact that the sink and the underlying elasticsearch  
library seems to keep a list of available IPs, the elastic load balancer  
just returns the ip address of an instance in a round robin way which the  
flume sink resolves and caches, so ultimately flume ends up with a fixed IP  
of one of the master instances for the entire duration of it's run. However  
this IP obviously changes everytime we redeploy our cluster.

So my questions are:

1. What is the best configuration for the master nodes? Should they have  
static IPs on AWS and flume should be configured to use these directly?  
Effectively ignoring the load balancer.
2. What do we in a multiple zone scenario? How would we handle failover?  
Should we have for example an extra set of static ips for each zone so that  
if a zone disappears the remaining zones will scale up and grab an ip from  
this list of static ips.

I suppose ultimately I'm trying to understand what the master node does and  
how it should be utilised in the amazon world.

Any help appreciated

Dip

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:09am UTC](https://discuss.elastic.co/t/flume-amazon-and-the-master-node/14197/2 "2017-07-06T02:09:33Z")

</div>

Closed on July 6, 2017, 2:09am UTC