Creating ES Cluster with dedicated masters

[animageofmine (Ankit Parikh)]

We are planning to create ES cluster with about 10 data nodes and 3 dedicated masters and we want to dockerize our cluster. Following is how we are planning to configure:

Let's say there are only 3 nodes in the cluster that includes 1 dedicated master (just to make this topic brief and easy to understand).

IP1: 172.30.84.10 (dedicated master)
IP2: 172.30.84.11
IP3: 172.30.84.12

Then we would use the dockerfile used by elasticsearch and run the following commands:

On host 172.30.84.10 (dedicated master)

docker run -d -p 9200:9200 -p 9300:9300 elasticsearch \
-Des.node.name="es1" \
-Des.cluster.name="mycluster" \
-Des.network.host=_eth0_ \
-Des.network.publish_host=172.30.84.10 \
-Des.discovery.zen.ping.unicast.hosts=172.30.84.10,172.30.84.11,172.30.84.12 \
-Des.discovery.zen.minimum_master_nodes=1 \
-Des.node.master=true \
-Des.node.data=false \
-Des.node.ingest=false

On host 172.30.84.11

docker run -d -p 9200:9200 -p 9300:9300 elasticsearch \
-Des.node.name="es2" \
-Des.cluster.name="mycluster" \
-Des.network.host=_eth0_ \
-Des.network.publish_host=172.30.84.11 \
-Des.discovery.zen.ping.unicast.hosts=172.30.84.10,172.30.84.11,172.30.84.12 \
-Des.discovery.zen.minimum_master_nodes=1
-Des.node.master=false \
-Des.node.data=true

On host 172.30.84.12

docker run -d -p 9200:9200 -p 9300:9300 elasticsearch \
-Des.node.name="es3" \
-Des.cluster.name="mycluster" \
-Des.network.host=_eth0_ \
-Des.network.publish_host=172.30.84.12 \
-Des.discovery.zen.ping.unicast.hosts=172.30.84.10,172.30.84.11,172.30.84.12 \
-Des.discovery.zen.minimum_master_nodes=1
-Des.node.master=false \
-Des.node.data=true

I understand that setting one dedicated master is fatal in case it goes down, but this is just for this example, we would set up at least 3 (odd number). Now, let's say all nodes are up and running. Following are my questions:

1. If we use JAVA ES native client to communicate with ES cluster, should all the queries route through only master or nodes? Or should we register all the IP addresses like the following:

   TransportClient client = new PreBuiltTransportClient(Settings.EMPTY)
   .addTransportAddress(new InetSocketTransportAddress("172.30.84.10", 9300))
   .addTransportAddress(new InetSocketTransportAddress("172.30.84.11", 9300));
   .addTransportAddress(new InetSocketTransportAddress("172.30.84.12", 9300));

2. If we should use all IPs, all of these need to be public facing IPs (may be). If not, which IPs should be used in java client?

3. If one of the nodes go down, do we need to update the code/configuration/ip addresses? Is there a better way to handle this? I am sure I am missing something basic, because hard-coding or even configuring the IPs seem like a bad idea (I am not network guy).

4. Do I have to set any other configurations like Gateway settings

Thank you for your help in advance.

[warkolm (Mark Walkom)]

1. Data or client, never dedicated masters.
2. Whatever is routable in your environment.
3. Just replace the node but use the same address. Otherwise use DNS. Or yes, you will need to update things.
4. Ideally, yes.

[animageofmine (Ankit Parikh)]

Great, thank you so much.

1. If sniff is enabled (client.transport.sniff), do I still need to specify all IPs? My understanding is that sniff will allow the client to get all the other nodes in the network if new nodes were added at all.
2. For answer of my 4th question, you mentioned ideally yes. Does this mean the settings are optional? Would appreciate a bit verbose response here.

Is there a sample setup that I can reference? Its kind of hard to find anywhere on the web, googled quite a bit.

[warkolm (Mark Walkom)]

Sniff is just a seed, so it'll get any other nodes in the cluster.

It's always a good idea to add settings that can improve recovery speeds. However these are set in the config file which means if you add nodes you need to alter them! You can also set these via the APIs.

[animageofmine (Ankit Parikh)]

An example or verbose reply would help. I guess a lot of things that are implicit to you are not implicit to me

[warkolm (Mark Walkom)]

Ahh sorry, these were updatable via the APIs in 2.X, not in 5.
See https://www.elastic.co/guide/en/elasticsearch/reference/5.0/modules-gateway.html

[animageofmine (Ankit Parikh)]

We are planning to use 2.4 most likely until version 5.0 is fully baked. Are these configs optional or mandatory?

[warkolm (Mark Walkom)]

Optional.

[animageofmine (Ankit Parikh)]

Thank you!