Question about unicast discovery that finds only 2 out of 3 active nodes

I am trying to set up a 3-node unicast-discovery cluster named
test-server-3amigos. But all I can get is 2 nodes that like each other and
a third node that appears to be a rejected lonely outcast.

I would have thought that the -D options would have overridden the cluster
to recognize all three nodes as part of the cluster.

Here are the command-line java arguments when elasticsearch is started.
Note that I set the node names to the host names for easier investigation!

ElasticSearch pid is 23436
/usr/bin/pargs
23436: /bin/java -server -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true
-XX:+UsePar
argv[0]: /bin/java
argv[1]: -server
argv[2]: -Xms256m
argv[3]: -Xmx1g
argv[4]: -Xss256k
argv[5]: -Djava.awt.headless=true
argv[6]: -XX:+UseParNewGC
argv[7]: -XX:+UseConcMarkSweepGC
argv[8]: -XX:CMSInitiatingOccupancyFraction=75
argv[9]: -XX:+UseCMSInitiatingOccupancyOnly
argv[10]: -XX:+HeapDumpOnOutOfMemoryError
argv[11]: -Delasticsearch
argv[12]: -Des.pidfile=/tmp/es-test-server-3amigos.pid
argv[13]:
-Des.path.home=/opt/test-server/versions/2781.116/elasticsearch-0.20.4
argv[14]: -cp
argv[15]:
/opt/test-server/current/elasticsearch-0.20.4/lib:/opt/test-server/versions/2781.116/elasticsearch-0.20.4/lib/elasticsearch-0.20.4.jar:/opt/test-server/versions/2781.116/elasticsearch-0.20.4/lib/:/opt/test-server/versions/2781.116/elasticsearch-0.20.4/lib/sigar/
argv[16]: -Des.node.name=testdev12
argv[17]: -Des.path.conf=/opt/test-server/current/config
argv[18]: -Ddiscovery.zen.ping.multicast.enabled=false
argv[19]: -Ddiscovery.zen.ping.unicast.hosts=testdev12,testdev29,testdev33
argv[20]: -Ddiscovery.zen.minimum_master_nodes=2
argv[21]: org.elasticsearch.bootstrap.ElasticSearch

And my elasticsearch.yml file is unchanged. Don't the command-line settings
above override the discovery? They seem to be ignored.

################################## Discovery
##################################

Discovery infrastructure ensures nodes can be found within a cluster

and master node is elected. Multicast discovery is the default.

Set to ensure a node sees N other master eligible nodes to be considered

operational within the cluster. Set this option to a higher value (2-4)

for large clusters (>3 nodes):

discovery.zen.minimum_master_nodes: 1

Set the time to wait for ping responses from other nodes when discovering.

Set this option to a higher value on a slow or congested network

to minimize discovery failures:

discovery.zen.ping.timeout: 3s

See http://elasticsearch.org/guide/reference/modules/discovery/zen.html

for more information.

Unicast discovery allows to explicitly control which nodes will be used

to discover the cluster. It can be used when multicast is not present,

or to restrict the cluster communication-wise.

1. Disable multicast discovery (enabled by default):

discovery.zen.ping.multicast.enabled: false

2. Configure an initial list of master nodes in the cluster

to perform discovery when new nodes (master or data) are started:

discovery.zen.ping.unicast.hosts: ["host1", "host2:port",

"host3[portX-portY]"]

Nodes 12 and 29 see each other but not node 33:

$ curl 'http://testdev12:9200/_cluster/state?pretty=true'
{
"cluster_name" : "test-lidb-3amigos",
"master_node" : "o5jRwGzxSqGYqgcl2aY-oA",
"blocks" : { },
"nodes" : {
"o5jRwGzxSqGYqgcl2aY-oA" : {
"name" : "testdev12",
"transport_address" : "inet[/192.168.200.222:9300]",
"attributes" : {
"max_local_storage_nodes" : "1"
}
},
"o2gtWBcqQB2-Tx22WwH2vg" : {
"name" : "testdev29",
"transport_address" : "inet[/192.168.200.241:9300]",
"attributes" : {
"max_local_storage_nodes" : "1"
}
}
},
"metadata" : {
"templates" : { },
"indices" : { }
},
"routing_table" : {
"indices" : { }
},
"routing_nodes" : {
"unassigned" : [ ],
"nodes" : {
"o5jRwGzxSqGYqgcl2aY-oA" : [ ],
"o2gtWBcqQB2-Tx22WwH2vg" : [ ]
}
},
"allocations" : [ ]
}

While node 33 is all by itself, but it shows up in ElasticSearch Head in a
green status:

$ curl 'http://testdev33:9200/_cluster/state?pretty=true'
{
"cluster_name" : "test-lidb-3amigos",
"master_node" : "cim4vJBRS3ajkCRD8FGIug",
"blocks" : { },
"nodes" : {
"cim4vJBRS3ajkCRD8FGIug" : {
"name" : "testdev33",
"transport_address" : "inet[/192.168.200.133:9300]",
"attributes" : {
"max_local_storage_nodes" : "1"
}
}
},
"metadata" : {
"templates" : { },
"indices" : { }
},
"routing_table" : {
"indices" : { }
},
"routing_nodes" : {
"unassigned" : [ ],
"nodes" : {
"cim4vJBRS3ajkCRD8FGIug" : [ ]
}
},
"allocations" : [ ]
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Prefix it with "es."

So it should be:
argv[18]: -Des.discovery.zen.ping.multicast.enabled=false
argv[19]: -Des.discovery.zen.ping.unicast.hosts=testdev12,testdev29,testdev33
argv[20]: -Des.discovery.zen.minimum_master_nodes=2

My 2 cents

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 avr. 2013 à 21:02, InquiringMind brian.from.fl@gmail.com a écrit :

I am trying to set up a 3-node unicast-discovery cluster named test-server-3amigos. But all I can get is 2 nodes that like each other and a third node that appears to be a rejected lonely outcast.

I would have thought that the -D options would have overridden the cluster to recognize all three nodes as part of the cluster.

Here are the command-line java arguments when elasticsearch is started. Note that I set the node names to the host names for easier investigation!

ElasticSearch pid is 23436
/usr/bin/pargs
23436: /bin/java -server -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UsePar
argv[0]: /bin/java
argv[1]: -server
argv[2]: -Xms256m
argv[3]: -Xmx1g
argv[4]: -Xss256k
argv[5]: -Djava.awt.headless=true
argv[6]: -XX:+UseParNewGC
argv[7]: -XX:+UseConcMarkSweepGC
argv[8]: -XX:CMSInitiatingOccupancyFraction=75
argv[9]: -XX:+UseCMSInitiatingOccupancyOnly
argv[10]: -XX:+HeapDumpOnOutOfMemoryError
argv[11]: -Delasticsearch
argv[12]: -Des.pidfile=/tmp/es-test-server-3amigos.pid
argv[13]: -Des.path.home=/opt/test-server/versions/2781.116/elasticsearch-0.20.4
argv[14]: -cp
argv[15]: /opt/test-server/current/elasticsearch-0.20.4/lib:/opt/test-server/versions/2781.116/elasticsearch-0.20.4/lib/elasticsearch-0.20.4.jar:/opt/test-server/versions/2781.116/elasticsearch-0.20.4/lib/:/opt/test-server/versions/2781.116/elasticsearch-0.20.4/lib/sigar/
argv[16]: -Des.node.name=testdev12
argv[17]: -Des.path.conf=/opt/test-server/current/config
argv[18]: -Ddiscovery.zen.ping.multicast.enabled=false
argv[19]: -Ddiscovery.zen.ping.unicast.hosts=testdev12,testdev29,testdev33
argv[20]: -Ddiscovery.zen.minimum_master_nodes=2
argv[21]: org.elasticsearch.bootstrap.ElasticSearch

And my elasticsearch.yml file is unchanged. Don't the command-line settings above override the discovery? They seem to be ignored.

################################## Discovery ##################################

Discovery infrastructure ensures nodes can be found within a cluster

and master node is elected. Multicast discovery is the default.

Set to ensure a node sees N other master eligible nodes to be considered

operational within the cluster. Set this option to a higher value (2-4)

for large clusters (>3 nodes):

discovery.zen.minimum_master_nodes: 1

Set the time to wait for ping responses from other nodes when discovering.

Set this option to a higher value on a slow or congested network

to minimize discovery failures:

discovery.zen.ping.timeout: 3s

See http://elasticsearch.org/guide/reference/modules/discovery/zen.html

for more information.

Unicast discovery allows to explicitly control which nodes will be used

to discover the cluster. It can be used when multicast is not present,

or to restrict the cluster communication-wise.

1. Disable multicast discovery (enabled by default):

discovery.zen.ping.multicast.enabled: false

2. Configure an initial list of master nodes in the cluster

to perform discovery when new nodes (master or data) are started:

discovery.zen.ping.unicast.hosts: ["host1", "host2:port", "host3[portX-portY]"]

Nodes 12 and 29 see each other but not node 33:

$ curl 'http://testdev12:9200/_cluster/state?pretty=true'
{
"cluster_name" : "test-lidb-3amigos",
"master_node" : "o5jRwGzxSqGYqgcl2aY-oA",
"blocks" : { },
"nodes" : {
"o5jRwGzxSqGYqgcl2aY-oA" : {
"name" : "testdev12",
"transport_address" : "inet[/192.168.200.222:9300]",
"attributes" : {
"max_local_storage_nodes" : "1"
}
},
"o2gtWBcqQB2-Tx22WwH2vg" : {
"name" : "testdev29",
"transport_address" : "inet[/192.168.200.241:9300]",
"attributes" : {
"max_local_storage_nodes" : "1"
}
}
},
"metadata" : {
"templates" : { },
"indices" : { }
},
"routing_table" : {
"indices" : { }
},
"routing_nodes" : {
"unassigned" : [ ],
"nodes" : {
"o5jRwGzxSqGYqgcl2aY-oA" : [ ],
"o2gtWBcqQB2-Tx22WwH2vg" : [ ]
}
},
"allocations" : [ ]
}

While node 33 is all by itself, but it shows up in ElasticSearch Head in a green status:

$ curl 'http://testdev33:9200/_cluster/state?pretty=true'
{
"cluster_name" : "test-lidb-3amigos",
"master_node" : "cim4vJBRS3ajkCRD8FGIug",
"blocks" : { },
"nodes" : {
"cim4vJBRS3ajkCRD8FGIug" : {
"name" : "testdev33",
"transport_address" : "inet[/192.168.200.133:9300]",
"attributes" : {
"max_local_storage_nodes" : "1"
}
}
},
"metadata" : {
"templates" : { },
"indices" : { }
},
"routing_table" : {
"indices" : { }
},
"routing_nodes" : {
"unassigned" : [ ],
"nodes" : {
"cim4vJBRS3ajkCRD8FGIug" : [ ]
}
},
"allocations" : [ ]
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

David,

Thank you very much. Early indications show that's exactly what was
missing. Your 2 cents apparently has a very favorable exchange rate! :slight_smile:

On Monday, April 29, 2013 3:33:36 PM UTC-4, David Pilato wrote:

Prefix it with "es."

So it should be:
argv[18]: -Des.discovery.zen.ping.multicast.enabled=false
argv[19]: *
-Des.discovery.zen.ping.unicast.hosts=testdev12,testdev29,testdev33*
argv[20]: -Des.discovery.zen.minimum_master_nodes=2

My 2 cents

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.