Explicit cluster configuration

Elastic Stack Elasticsearch
1

I'm looking for docs on how to explicitly define my cluster. What I
want is something like zookeeper's config file (a sample is pasted in
at the bottom of this email), I don't want elasticsearch doing
automatic node discovery, and I especially want it to report to me
that nodes are missing when it can't contact them. As it's running
right now, every zookeeper node that I start on my network (regardless
of what I specify in -Des.cluster.name) joins the existing
elasticsearch cluster, which always has a green status. I thought I
had figured out how to disable the automatic discovery in the
elasticsearch.json, but it seems to have not worked. My attempt is
below; I'm hoping somebody can show me what I did wrong, and point me
in the correct direction.

my attempt at explicitly configuring a cluster:
{
"discovery": {
"zen": {
"ping": {
"unicast": {
"hosts": [
"kvm-alpha",
"kvm-beta"
]
}
}
}
}
}

sample zookeeper.config:

tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888

2

The master config file comes with (commented out) example of how to do unicast discovery (and disabling multicast): https://github.com/elasticsearch/elasticsearch/blob/master/config/elasticsearch.yml.

On Thursday, June 16, 2011 at 5:09 AM, tsuraan wrote:

I'm looking for docs on how to explicitly define my cluster. What I
want is something like zookeeper's config file (a sample is pasted in
at the bottom of this email), I don't want elasticsearch doing
automatic node discovery, and I especially want it to report to me
that nodes are missing when it can't contact them. As it's running
right now, every zookeeper node that I start on my network (regardless
of what I specify in -Des.cluster.name (http://-Des.cluster.name)) joins the existing
elasticsearch cluster, which always has a green status. I thought I
had figured out how to disable the automatic discovery in the
elasticsearch.json, but it seems to have not worked. My attempt is
below; I'm hoping somebody can show me what I did wrong, and point me
in the correct direction.

my attempt at explicitly configuring a cluster:
{
"discovery": {
"zen": {
"ping": {
"unicast": {
"hosts": [
"kvm-alpha",
"kvm-beta"
]
}
}
}
}
}

sample zookeeper.config:

tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888

3

The master config file comes with (commented out) example of how to do
unicast discovery (and disabling
multicast): https://github.com/elasticsearch/elasticsearch/blob/master/config/elasticsearch.yml.

Ok, I'm trying that with the ping.unicast.hosts set to
["kvm-alpha:9300", "kvm-beta:9300", "kvm-gamma:9300"] and
elasticsearch running at version 0.16.2. Only kvm-alpha and kvm-gamma
are running, so those are the only two machines where Elasticsearch
is currently running. It's behaving differently now, anyhow.
/_cluster/state for each machine lists only the elasticsearch running
on that machine. netstat shows that both machine's elasticsearch's
have connections to port 9300 of the other machine, so they are in
contact, but the don't seem to quite be meshing. A pretty dump of
/_cluster/state is at the bottom of this email. /_cluster/health now
shows red; I'm not sure what to make of the rest of its values though.
I definitely have no idea if it's red because it realizes that nodes
are missing, or for some other reason. I'm guessing that the only
reason ES was working at all before was due to the multicast cluster
configuration taking over, so now I have something just plain wrong.

For basic diagnostics, each machine can ping each other by short name.
/var/log/elasticsearch/blah.log has a lot of
java.net.NoRouteToHostExceptions; I'm assuming these are due to the
kvm-beta host of my unicast hosts, but the traceback doesn't actually
say which host has no route, so I can't be sure. Other than that,
I've wiped out /var/lib/elasticsearch/data/* and
/var/lib/elasticsearch/work/*; not sure what else I can do.

/_cluster/state on kvm-gamma:

{
"cluster_name" : "blah",
"master_node" : "Isx1p9MgS3OKxfrt_OPKcg",
"blocks" : {
"global" : {
"1" : {
"description" : "state not recovered / initialized",
"retryable" : true,
"disable_state_persistence" : true,
"levels" : [ "read", "write", "metadata" ]
}
}
},
"nodes" : {
"Isx1p9MgS3OKxfrt_OPKcg" : {
"name" : "American Dream",
"transport_address" : "inet[/10.77.97.31:9300]",
"attributes" : {
}
}
},
"metadata" : {
"templates" : {
},
"indices" : {
}
},
"routing_table" : {
"indices" : {
}
},
"routing_nodes" : {
"unassigned" : ,
"nodes" : {
}
},
"allocations" :
}

/_cluster/info on kvm-gamma

{
"cluster_name" : "blah",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

and my elasticsearch.yml:

Cluster Settings

cluster:
name: blah

Path Settings

path:
conf: /usr/share/elasticsearch/conf
data: /var/lib/elasticsearch/data
work: /var/lib/elasticsearch/work
logs: /var/log/elasticsearch

Gateway Settings

gateway:
recover_after_nodes: 1
recover_after_time: 5m
expected_nodes: 2

Force all memory to be locked, forcing JVM to never swap

(make sure to set MIN and MAX mem to the same value)

#bootstrap:

mlockall: true

Unicast Discovery (disable multicast)

discovery:
zen:
multicast.enabled: false
ping.unicast.hosts: ["kvm-alpha:9300", "kvm-beta:9300", "kvm-gamma:9300"]

4

First, make sure to disable multicast as well (as shown in the example).

The no route to host means that the host names can't be resolved. Try first with IPs, see if it gets solved.

On Thursday, June 16, 2011 at 6:53 PM, tsuraan wrote:

The master config file comes with (commented out) example of how to do
unicast discovery (and disabling
multicast): https://github.com/elasticsearch/elasticsearch/blob/master/config/elasticsearch.yml.

Ok, I'm trying that with the ping.unicast.hosts set to
["kvm-alpha:9300", "kvm-beta:9300", "kvm-gamma:9300"] and
elasticsearch running at version 0.16.2. Only kvm-alpha and kvm-gamma
are running, so those are the only two machines where Elasticsearch
is currently running. It's behaving differently now, anyhow.
/_cluster/state for each machine lists only the elasticsearch running
on that machine. netstat shows that both machine's elasticsearch's
have connections to port 9300 of the other machine, so they are in
contact, but the don't seem to quite be meshing. A pretty dump of
/_cluster/state is at the bottom of this email. /_cluster/health now
shows red; I'm not sure what to make of the rest of its values though.
I definitely have no idea if it's red because it realizes that nodes
are missing, or for some other reason. I'm guessing that the only
reason ES was working at all before was due to the multicast cluster
configuration taking over, so now I have something just plain wrong.

For basic diagnostics, each machine can ping each other by short name.
/var/log/elasticsearch/blah.log has a lot of
java.net.NoRouteToHostExceptions; I'm assuming these are due to the
kvm-beta host of my unicast hosts, but the traceback doesn't actually
say which host has no route, so I can't be sure. Other than that,
I've wiped out /var/lib/elasticsearch/data/* and
/var/lib/elasticsearch/work/*; not sure what else I can do.

/_cluster/state on kvm-gamma:

{
"cluster_name" : "blah",
"master_node" : "Isx1p9MgS3OKxfrt_OPKcg",
"blocks" : {
"global" : {
"1" : {
"description" : "state not recovered / initialized",
"retryable" : true,
"disable_state_persistence" : true,
"levels" : [ "read", "write", "metadata" ]
}
}
},
"nodes" : {
"Isx1p9MgS3OKxfrt_OPKcg" : {
"name" : "American Dream",
"transport_address" : "inet[/10.77.97.31:9300]",
"attributes" : {
}
}
},
"metadata" : {
"templates" : {
},
"indices" : {
}
},
"routing_table" : {
"indices" : {
}
},
"routing_nodes" : {
"unassigned" : ,
"nodes" : {
}
},
"allocations" :
}

/_cluster/info on kvm-gamma

{
"cluster_name" : "blah",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

and my elasticsearch.yml:

Cluster Settings

cluster:
name: blah

Path Settings

path:
conf: /usr/share/elasticsearch/conf
data: /var/lib/elasticsearch/data
work: /var/lib/elasticsearch/work
logs: /var/log/elasticsearch

Gateway Settings

gateway:
recover_after_nodes: 1
recover_after_time: 5m
expected_nodes: 2

Force all memory to be locked, forcing JVM to never swap

(make sure to set MIN and MAX mem to the same value)

#bootstrap:

mlockall: true

Unicast Discovery (disable multicast)

discovery:
zen:
multicast.enabled: false
ping.unicast.hosts: ["kvm-alpha:9300", "kvm-beta:9300", "kvm-gamma:9300"]

5

I have the similar problem.
There is the part of my master config file(elasticsearch.yml):

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["xxx.xx.168.10:9300", "xxx.xx.12.85:9300"]
discovery.zen.ping.timeout: 10s
discovery.zen.minimum_master_nodes: 2

The problem is that xxx.xx.12.85:9300 is found but "xxx.xx.168.10:9300" is not.
Master IP xxx.xx.12.128.

Any ideas?

Thanks.