Discovery.zen.minimum_master_nodes and elasticsearch_config.yml

Elastic Stack Elasticsearch
1

Is the config file referenced for discovery.zen.minimum_master_nodes or is
only the state in the cluster used ?

This is the master starting up:

[2013-12-11 06:02:07,519][INFO ][bootstrap ] max_open_files
[99970]

[2013-12-11 06:02:07,587][INFO ][node ] [Black Tom
Cassidy] version[0.90.7], pid[22299], build[36897d0/2013-11-13T12:06:54Z]

[2013-12-11 06:02:07,587][INFO ][node ] [Black Tom
Cassidy] initializing ...

[2013-12-11 06:02:07,591][INFO ][plugins ] [Black Tom
Cassidy] loaded [], sites []

[2013-12-11 06:02:08,876][DEBUG][discovery.zen.ping.multicast] [Black Tom
Cassidy] using group [224.2.2.4], with port [54328], ttl [3], and address
[null]

[2013-12-11 06:02:08,879][DEBUG][discovery.zen.ping.unicast] [Black Tom
Cassidy] using initial hosts [], with concurrent_connects [10]

[2013-12-11 06:02:08,879][DEBUG][discovery.zen ] [Black Tom
Cassidy] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.elect ] [Black Tom
Cassidy] using minimum_master_nodes [-1]

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.fd ] [Black Tom
Cassidy] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries
[3]

[2013-12-11 06:02:08,883][DEBUG][discovery.zen.fd ] [Black Tom
Cassidy] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries
[3]

curl -XGET localhost:9200/_cluster/settings?pretty=true

{

"persistent" : { },

"transient" : { }

}

Our config looks like:

cluster.name: ao3elastic

node.data: false

node.master: false

index.number_of_shards: 5

index.number_of_replicas: 1

path.conf: /home/ao3elastic/conf

path.data: /home/ao3elastic/data

path.work: /home/ao3elastic/tmp

path.logs: /home/ao3elastic/log

path.plugins: /home/ao3elastic/plugin

network.host: 0.0.0.0

http.enabled: true

gateway.recover_after_nodes: 1

gateway.expected_nodes: 1

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.timeout: 5s

monitor.jvm.gc.ParNew.warn: 1000ms

monitor.jvm.gc.ParNew.info: 700ms

monitor.jvm.gc.ParNew.debug: 400ms

monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s

monitor.jvm.gc.ConcurrentMarkSweep.info: 5s

monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4c3dac3d-2c94-48b7-8301-52f732aa2b03%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi James,

This is tricky. During normal running of the cluster, the cluster state
setting are used, but if you restart a node both play a role. In short -
you use the elasticsearch.yml to control initial joining of the cluster and
the cluster state one to introduce changes later on. To keep things simple,
I would use the one in the config file for most purposes and use the
transient version of the cluster state for when you need temporary changes
due to administration work.

This is how it works:

  1. When a node starts, it reads it's local elasticsearch.yml settings in
    order to know how to join the cluster (for example, read the cluster name)
  2. The node will try to join the cluster and validate that the current
    cluster it sees conforms to the requirements in it's config file. If not it
    will continue pinging.
  3. If there is already an active master in the cluster - the node will
    accept it and update it's local setting from the master's version of the
    cluster state (now it gets the new minimum master node settings if set).
  4. If there is no master, the node will go into a master election with the
    other nodes. The winning master will read the cluster state from it's local
    persisted settings and if it contains a minimum master node settings, this
    will be communicated to the rest of the cluster.

Cheers,
Boaz

On Wednesday, December 11, 2013 11:54:37 PM UTC+1, James Beal wrote:

Is the config file referenced for discovery.zen.minimum_master_nodes or is
only the state in the cluster used ?

This is the master starting up:

[2013-12-11 06:02:07,519][INFO ][bootstrap ] max_open_files
[99970]

[2013-12-11 06:02:07,587][INFO ][node ] [Black Tom
Cassidy] version[0.90.7], pid[22299], build[36897d0/2013-11-13T12:06:54Z]

[2013-12-11 06:02:07,587][INFO ][node ] [Black Tom
Cassidy] initializing ...

[2013-12-11 06:02:07,591][INFO ][plugins ] [Black Tom
Cassidy] loaded , sites

[2013-12-11 06:02:08,876][DEBUG][discovery.zen.ping.multicast] [Black Tom
Cassidy] using group [224.2.2.4], with port [54328], ttl [3], and address
[null]

[2013-12-11 06:02:08,879][DEBUG][discovery.zen.ping.unicast] [Black Tom
Cassidy] using initial hosts , with concurrent_connects [10]

[2013-12-11 06:02:08,879][DEBUG][discovery.zen ] [Black Tom
Cassidy] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.elect ] [Black Tom
Cassidy] using minimum_master_nodes [-1]

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.fd ] [Black Tom
Cassidy] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries
[3]

[2013-12-11 06:02:08,883][DEBUG][discovery.zen.fd ] [Black Tom
Cassidy] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries
[3]

curl -XGET localhost:9200/_cluster/settings?pretty=true

{

"persistent" : { },

"transient" : { }

}

Our config looks like:

cluster.name: ao3elastic

node.data: false

node.master: false

index.number_of_shards: 5

index.number_of_replicas: 1

path.conf: /home/ao3elastic/conf

path.data: /home/ao3elastic/data

path.work: /home/ao3elastic/tmp

path.logs: /home/ao3elastic/log

path.plugins: /home/ao3elastic/plugin

network.host: 0.0.0.0

http.enabled: true

gateway.recover_after_nodes: 1

gateway.expected_nodes: 1

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.timeout: 5s

monitor.jvm.gc.ParNew.warn: 1000ms

monitor.jvm.gc.ParNew.info: 700ms

monitor.jvm.gc.ParNew.debug: 400ms

monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s

monitor.jvm.gc.ConcurrentMarkSweep.info: 5s

monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb333c50-985e-45c1-a490-bdc6888bbd07%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

On Sunday, December 15, 2013 9:52:34 AM UTC, Boaz Leskes wrote:

Hi James,

This is tricky. During normal running of the cluster, the cluster state
setting are used, but if you restart a node both play a role. In short -
you use the elasticsearch.yml to control initial joining of the cluster and
the cluster state one to introduce changes later on. To keep things simple,
I would use the one in the config file for most purposes and use the
transient version of the cluster state for when you need temporary changes
due to administration work.

This is how it works:

  1. When a node starts, it reads it's local elasticsearch.yml settings in
    order to know how to join the cluster (for example, read the cluster name)
  2. The node will try to join the cluster and validate that the current
    cluster it sees conforms to the requirements in it's config file. If not it
    will continue pinging.
  3. If there is already an active master in the cluster - the node will
    accept it and update it's local setting from the master's version of the
    cluster state (now it gets the new minimum master node settings if set).
  4. If there is no master, the node will go into a master election with the
    other nodes. The winning master will read the cluster state from it's local
    persisted settings and if it contains a minimum master node settings, this
    will be communicated to the rest of the cluster.

Cheers,
Boaz

Thank you for your reply it is very much appreciated.

Given that I have a cluster running with a single master eligible node and
"discovery.zen.minimum_master_nodes: 2" configured in elasticsearch.yml It
seems apparent that I should also set this in the cluster state, should it
be in the transient or persistent state ( my intuition says it should be in
the persistent state ).

What I think you are saying is that the reason for the master starting was
that the local persistent state did not have discovery.zen.minimum_master_nodes
configured and the local persistent state overruled the config file.

To get to a cluster with multiple master eligible nodes I think the correct
set of actions would be:

  • Add two master eligible nodes
  • Set the discovery.zen.minimum_master_nodes to two in the persistent state.

curl -XPUT localhost:9200/_cluster/settings -d '{ "persistent" :
{"discovery.zen.minimum_master_nodes" : 2 }}'

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/490e6e38-07f6-4d92-ab9e-4ed571c22221%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi James,

Configuring the elasticsearch.yml is enough. I think your settings
"discovery.zen.minimum_master_nodes: 2" in your elasticsearch.yml are not
picked up - see log:

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.elect ] [Black Tom
Cassidy] using minimum_master_nodes [-1]

Also if you have it set to 2 and you only have a single master - the node
should not start but start pinging searching for another
master eligible node...

On Sunday, December 15, 2013 2:36:45 PM UTC+1, James Beal wrote:

On Sunday, December 15, 2013 9:52:34 AM UTC, Boaz Leskes wrote:

Hi James,

This is tricky. During normal running of the cluster, the cluster state
setting are used, but if you restart a node both play a role. In short -
you use the elasticsearch.yml to control initial joining of the cluster and
the cluster state one to introduce changes later on. To keep things simple,
I would use the one in the config file for most purposes and use the
transient version of the cluster state for when you need temporary changes
due to administration work.

This is how it works:

  1. When a node starts, it reads it's local elasticsearch.yml settings in
    order to know how to join the cluster (for example, read the cluster name)
  2. The node will try to join the cluster and validate that the current
    cluster it sees conforms to the requirements in it's config file. If not it
    will continue pinging.
  3. If there is already an active master in the cluster - the node will
    accept it and update it's local setting from the master's version of the
    cluster state (now it gets the new minimum master node settings if set).
  4. If there is no master, the node will go into a master election with
    the other nodes. The winning master will read the cluster state from it's
    local persisted settings and if it contains a minimum master node settings,
    this will be communicated to the rest of the cluster.

Cheers,
Boaz

Thank you for your reply it is very much appreciated.

Given that I have a cluster running with a single master eligible node
and "discovery.zen.minimum_master_nodes: 2" configured in
elasticsearch.yml It seems apparent that I should also set this in the
cluster state, should it be in the transient or persistent state ( my
intuition says it should be in the persistent state ).

What I think you are saying is that the reason for the master starting was
that the local persistent state did not have discovery.zen.minimum_master_nodes
configured and the local persistent state overruled the config file.

To get to a cluster with multiple master eligible nodes I think the
correct set of actions would be:

  • Add two master eligible nodes
  • Set the discovery.zen.minimum_master_nodes to two in the persistent
    state.

curl -XPUT localhost:9200/_cluster/settings -d '{ "persistent" :
{"discovery.zen.minimum_master_nodes" : 2 }}'

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/479d44fa-eb15-419f-add0-3961eef24676%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

On Sunday, December 15, 2013 9:34:04 PM UTC, Boaz Leskes wrote:

Hi James,

Configuring the elasticsearch.yml is enough. I think your settings
"discovery.zen.minimum_master_nodes: 2" in your elasticsearch.yml are not
picked up - see log:

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.elect ] [Black Tom
Cassidy] using minimum_master_nodes [-1]

Also if you have it set to 2 and you only have a single master - the node
should not start but start pinging searching for another
master eligible node...

This is what I would have expected from the documentation however the node
started happily...., for example on my test Elasticsearch cluster...

root@test-elastic01:/home/ao3elastic/log# grep
"discovery.zen.minimum_master_nodes:"
/home/ao3elastic/conf/elasticsearch_config.yml

discovery.zen.minimum_master_nodes: 1

discovery.zen.minimum_master_nodes: 2

root@test-elastic01:/home/ao3elastic/log# /etc/init.d/ao3elastic start

[ ok ] Starting Elasticsearch ao3elastic Server:.

root@test-elastic01:/home/ao3elastic/log# cat ao3elastic.log

[2013-12-15 22:39:34,367][INFO ][bootstrap ] max_open_files
[99967]

[2013-12-15 22:39:34,515][INFO ][node ] [Astaroth /
Asteroth] version[0.90.7], pid[26892], build[36897d0/2013-11-13T12:06:54Z]

[2013-12-15 22:39:34,515][INFO ][node ] [Astaroth /
Asteroth] initializing ...

[2013-12-15 22:39:34,522][INFO ][plugins ] [Astaroth /
Asteroth] loaded , sites

[2013-12-15 22:39:36,374][DEBUG][discovery.zen.ping.multicast] [Astaroth /
Asteroth] using group [224.2.2.4], with port [54328], ttl [3], and address
[null]

[2013-12-15 22:39:36,378][DEBUG][discovery.zen.ping.unicast] [Astaroth /
Asteroth] using initial hosts , with concurrent_connects [10]

[2013-12-15 22:39:36,380][DEBUG][discovery.zen ] [Astaroth /
Asteroth] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]

[2013-12-15 22:39:36,381][DEBUG][discovery.zen.elect ] [Astaroth /
Asteroth] using minimum_master_nodes [-1]

[2013-12-15 22:39:36,382][DEBUG][discovery.zen.fd ] [Astaroth /
Asteroth] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]

[2013-12-15 22:39:36,386][DEBUG][discovery.zen.fd ] [Astaroth /
Asteroth] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]

[2013-12-15 22:39:37,864][INFO ][node ] [Astaroth /
Asteroth] initialized

[2013-12-15 22:39:37,865][INFO ][node ] [Astaroth /
Asteroth] starting ...

[2013-12-15 22:39:37,999][INFO ][transport ] [Astaroth /
Asteroth] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.10.8.26:9300]}

[2013-12-15 22:39:38,022][TRACE][discovery ] [Astaroth /
Asteroth] waiting for 30s for the initial state to be set by the discovery

[2013-12-15 22:39:38,032][TRACE][discovery.zen.ping.multicast] [Astaroth /
Asteroth] [1] sending ping request

[2013-12-15 22:39:39,545][TRACE][discovery.zen.ping.multicast] [Astaroth /
Asteroth] [1] sending ping request

[2013-12-15 22:39:41,050][TRACE][discovery.zen ] [Astaroth /
Asteroth] full ping responses: {none}

[2013-12-15 22:39:41,051][DEBUG][discovery.zen ] [Astaroth /
Asteroth] filtered ping responses: (filter_client[true],
filter_data[false]) {none}

[2013-12-15 22:39:41,063][INFO ][cluster.service ] [Astaroth /
Asteroth] new_master [Astaroth /
Asteroth][hRTd6QwHTfGAwucuSkNBCA][inet[/10.10.8.26:9300]]{master=true},
reason: zen-disco-join (elected_as_master)

[2013-12-15 22:39:41,113][TRACE][discovery ] [Astaroth /
Asteroth] initial state set from discovery

[2013-12-15 22:39:41,113][INFO ][discovery ] [Astaroth /
Asteroth] ao3elastic/hRTd6QwHTfGAwucuSkNBCA

[2013-12-15 22:39:41,771][INFO ][http ] [Astaroth /
Asteroth] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.10.8.26:9200]}

[2013-12-15 22:39:41,771][INFO ][node ] [Astaroth /
Asteroth] started

[2013-12-15 22:39:45,305][INFO ][monitor.jvm ] [Astaroth /
Asteroth] [gc][ParNew][7][2] duration [785ms], collections [1]/[1.4s],
total [785ms]/[1.3s], memory [267.1mb]->[112.8mb]/[35.9gb], all_pools
{[Code Cache] [1.2mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[242.7mb]->[2.7mb]/[409.6mb]}{[Par Survivor Space]
[24.4mb]->[51.1mb]/[51.1mb]}{[CMS Old Gen] [0b]->[58.8mb]/[35.5gb]}{[CMS
Perm Gen] [29.2mb]->[29.3mb]/[166mb]}

root@test-elastic01:/home/ao3elastic/log# ps -ef | grep java

6008 26892 1 21 22:39 ? 00:00:17
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms36864m -Xmx36864m -Xmn512m
-Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Des.max-open-files=true -Delasticsearch
-Des.pidfile=/var/run/ao3elastic.pid
-Des.path.home=/usr/local/elasticsearch -cp
:/usr/local/elasticsearch/lib/elasticsearch-0.90.7.jar:/usr/local/elasticsearch/lib/:/usr/local/elasticsearch/lib/sigar/
-Des.node.data=true -Des.node.master=true
-Des.default.config=/home/ao3elastic/conf/elasticsearch_config.yml
-Des.default.path.home=/usr/local/elasticsearch/
-Des.default.path.logs=/home/ao3elastic/log
-Des.default.path.data=/home/ao3elastic/data
-Des.default.path.work=/home/ao3elastic/tmp
-Des.default.path.conf=/home/ao3elastic/conf
org.elasticsearch.bootstrap.Elasticsearch

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/57f14f51-bd47-45cb-a228-a41ae0afb764%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hi James,

Everything looks good to me, so I'm puzzled by the ignored settings. A
couple of questions:

  1. Did you use a package to install ES?
  2. can you check if there is another elasticsearch.yml
    in /home/ao3elastic/conf with a different setting?
  3. Are other settings from the elasticsearch_config.yml picked up? Did you
    change the cluster name?

Boaz

On Sunday, December 15, 2013 11:43:29 PM UTC+1, James Beal wrote:

On Sunday, December 15, 2013 9:34:04 PM UTC, Boaz Leskes wrote:

Hi James,

Configuring the elasticsearch.yml is enough. I think your settings
"discovery.zen.minimum_master_nodes: 2" in your elasticsearch.yml are not
picked up - see log:

[2013-12-11 06:02:08,880][DEBUG][discovery.zen.elect ] [Black Tom
Cassidy] using minimum_master_nodes [-1]

Also if you have it set to 2 and you only have a single master - the node
should not start but start pinging searching for another
master eligible node...

This is what I would have expected from the documentation however the node
started happily...., for example on my test Elasticsearch cluster...

root@test-elastic01:/home/ao3elastic/log# grep
"discovery.zen.minimum_master_nodes:"
/home/ao3elastic/conf/elasticsearch_config.yml

discovery.zen.minimum_master_nodes: 1

discovery.zen.minimum_master_nodes: 2

root@test-elastic01:/home/ao3elastic/log# /etc/init.d/ao3elastic start

[ ok ] Starting Elasticsearch ao3elastic Server:.

root@test-elastic01:/home/ao3elastic/log# cat ao3elastic.log

[2013-12-15 22:39:34,367][INFO ][bootstrap ] max_open_files
[99967]

[2013-12-15 22:39:34,515][INFO ][node ] [Astaroth /
Asteroth] version[0.90.7], pid[26892], build[36897d0/2013-11-13T12:06:54Z]

[2013-12-15 22:39:34,515][INFO ][node ] [Astaroth /
Asteroth] initializing ...

[2013-12-15 22:39:34,522][INFO ][plugins ] [Astaroth /
Asteroth] loaded , sites

[2013-12-15 22:39:36,374][DEBUG][discovery.zen.ping.multicast] [Astaroth /
Asteroth] using group [224.2.2.4], with port [54328], ttl [3], and address
[null]

[2013-12-15 22:39:36,378][DEBUG][discovery.zen.ping.unicast] [Astaroth /
Asteroth] using initial hosts , with concurrent_connects [10]

[2013-12-15 22:39:36,380][DEBUG][discovery.zen ] [Astaroth /
Asteroth] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]

[2013-12-15 22:39:36,381][DEBUG][discovery.zen.elect ] [Astaroth /
Asteroth] using minimum_master_nodes [-1]

[2013-12-15 22:39:36,382][DEBUG][discovery.zen.fd ] [Astaroth /
Asteroth] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]

[2013-12-15 22:39:36,386][DEBUG][discovery.zen.fd ] [Astaroth /
Asteroth] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]

[2013-12-15 22:39:37,864][INFO ][node ] [Astaroth /
Asteroth] initialized

[2013-12-15 22:39:37,865][INFO ][node ] [Astaroth /
Asteroth] starting ...

[2013-12-15 22:39:37,999][INFO ][transport ] [Astaroth /
Asteroth] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.10.8.26:9300]}

[2013-12-15 22:39:38,022][TRACE][discovery ] [Astaroth /
Asteroth] waiting for 30s for the initial state to be set by the discovery

[2013-12-15 22:39:38,032][TRACE][discovery.zen.ping.multicast] [Astaroth /
Asteroth] [1] sending ping request

[2013-12-15 22:39:39,545][TRACE][discovery.zen.ping.multicast] [Astaroth /
Asteroth] [1] sending ping request

[2013-12-15 22:39:41,050][TRACE][discovery.zen ] [Astaroth /
Asteroth] full ping responses: {none}

[2013-12-15 22:39:41,051][DEBUG][discovery.zen ] [Astaroth /
Asteroth] filtered ping responses: (filter_client[true],
filter_data[false]) {none}

[2013-12-15 22:39:41,063][INFO ][cluster.service ] [Astaroth /
Asteroth] new_master [Astaroth /
Asteroth][hRTd6QwHTfGAwucuSkNBCA][inet[/10.10.8.26:9300]]{master=true},
reason: zen-disco-join (elected_as_master)

[2013-12-15 22:39:41,113][TRACE][discovery ] [Astaroth /
Asteroth] initial state set from discovery

[2013-12-15 22:39:41,113][INFO ][discovery ] [Astaroth /
Asteroth] ao3elastic/hRTd6QwHTfGAwucuSkNBCA

[2013-12-15 22:39:41,771][INFO ][http ] [Astaroth /
Asteroth] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.10.8.26:9200]}

[2013-12-15 22:39:41,771][INFO ][node ] [Astaroth /
Asteroth] started

[2013-12-15 22:39:45,305][INFO ][monitor.jvm ] [Astaroth /
Asteroth] [gc][ParNew][7][2] duration [785ms], collections [1]/[1.4s],
total [785ms]/[1.3s], memory [267.1mb]->[112.8mb]/[35.9gb], all_pools
{[Code Cache] [1.2mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[242.7mb]->[2.7mb]/[409.6mb]}{[Par Survivor Space]
[24.4mb]->[51.1mb]/[51.1mb]}{[CMS Old Gen] [0b]->[58.8mb]/[35.5gb]}{[CMS
Perm Gen] [29.2mb]->[29.3mb]/[166mb]}

root@test-elastic01:/home/ao3elastic/log# ps -ef | grep java

6008 26892 1 21 22:39 ? 00:00:17
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms36864m -Xmx36864m -Xmn512m
-Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Des.max-open-files=true -Delasticsearch
-Des.pidfile=/var/run/ao3elastic.pid
-Des.path.home=/usr/local/elasticsearch -cp
:/usr/local/elasticsearch/lib/elasticsearch-0.90.7.jar:/usr/local/elasticsearch/lib/:/usr/local/elasticsearch/lib/sigar/
-Des.node.data=true -Des.node.master=true
-Des.default.config=/home/ao3elastic/conf/elasticsearch_config.yml
-Des.default.path.home=/usr/local/elasticsearch/
-Des.default.path.logs=/home/ao3elastic/log
-Des.default.path.data=/home/ao3elastic/data
-Des.default.path.work=/home/ao3elastic/tmp
-Des.default.path.conf=/home/ao3elastic/conf
org.elasticsearch.bootstrap.Elasticsearch

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9d707780-ddb2-4536-925d-ceb8dbca2046%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

I have a Bash script that wraps the elasticsearch script. I then have a
server properties file that this Bash script reads to specify (optionally)
the list of hosts in the cluster. The Bash script extracts the configured
list and stores it in the HOSTS shell variable.

If the list of comma-separated host names is not empty, then the script
processes it as follows, and the $OPTS are passed to the elasticsearchscript which has never missed yet in recognizing them and picking them up
properly.

if [[ ! -z "$HOSTS" ]]; then

The minimum number of masters should generally be N/2 + 1, where N is

the number of hosts. For an example, with 3 hosts there should be a

minimum of 2 masters.

Split the host name list on commas, then count the number of host

names. Divide by 2 (integer arithmetic; fractional remainder is

discarded) and add 1.

NHOSTS=$(echo $(echo $HOSTS | sed "s/[,]/ /g" | wc -w))
MIN_MASTERS=$((($NHOSTS / 2) + 1))
#echo "NHOSTS=$NHOSTS MIN_MASTERS=$MIN_MASTERS" >&2

OPTS="-Des.discovery.zen.ping.multicast.enabled=false"
OPTS="$OPTS -Des.discovery.zen.ping.unicast.hosts=$HOSTS"
OPTS="$OPTS -Des.discovery.zen.minimum_master_nodes=$MIN_MASTERS"
fi

I do this because I have a gaggle of test environments for all sorts of
uses, and I have one generic ES installation with one generic (and very
basic) elasticsearch.yml. Of course, I set up each index+type with its own
mappings, relying on neither the default mapping logic nor a pre-configured
default.

I hope this helps.

Regards,
Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31e5d286-9c35-4263-b6cb-fba1470df760%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

On Monday, December 16, 2013 12:30:39 PM UTC, Boaz Leskes wrote:

Hi James,

Everything looks good to me, so I'm puzzled by the ignored settings. A
couple of questions:

  1. Did you use a package to install ES?

No I pulled down the tar file and packaged it as I don't think originally
you did a deb file. I will have a look at the .deb file you have and see if
that would work for us.

  1. can you check if there is another elasticsearch.yml
    in /home/ao3elastic/conf with a different setting?

I can't see any other files...

root@test-elastic01:/home/james_# find /usr/otw /home/ao3elastic
/usr/local -name "elasticsearch_config.yml" -print

/home/ao3elastic/conf/elasticsearch_config.yml

  1. Are other settings from the elasticsearch_config.yml picked up? Did you
    change the cluster name?

Yes I changed the name and that took...

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "ao3elastic",

Boaz

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/50cafc60-1c6d-49ba-828d-ccc77c087c2e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

I can confirm that setting es.discovery.zen.minimum_master_nodes on the
command line looks as if it works unlike the setting in the config file.
Thank you Brian that looks like a workable work around.

I have additionally downloaded the deb file and the binaries and libs are
identical to my repackaged version

I have added the following to my defaults section and this looks like it
works.

Grab the number of hosts

HOSTS=grep discovery.zen.ping.unicast.hosts: $CONF_FILE | grep -v "^#" | awk -F: '{print $2}' | sed -e 's/\[//' -e 's/\]//'

NHOSTS=$(echo $(echo $HOSTS | sed "s/[,]/ /g" | wc -w))

MIN_MASTERS=$((($NHOSTS / 2) + 1))

Additional Java OPTS

ES_JAVA_OPTS="-Des.max-open-files=true
-Des.discovery.zen.minimum_master_nodes=$MIN_MASTERS"

And the log.

[2013-12-16 18:33:08,480][INFO ][bootstrap ] max_open_files
[99967]

[2013-12-16 18:33:08,617][INFO ][node ] [Batragon]
version[0.90.7], pid[1450], build[36897d0/2013-11-13T12:06:54Z]

[2013-12-16 18:33:08,617][INFO ][node ] [Batragon]
initializing ...

[2013-12-16 18:33:08,624][INFO ][plugins ] [Batragon]
loaded [], sites []

[2013-12-16 18:33:10,316][DEBUG][discovery.zen.ping.multicast] [Batragon]
using group [224.2.2.4], with port [54328], ttl [3], and address [null]

[2013-12-16 18:33:10,320][DEBUG][discovery.zen.ping.unicast] [Batragon]
using initial hosts [], with concurrent_connects [10]

[2013-12-16 18:33:10,321][DEBUG][discovery.zen ] [Batragon]
using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]

[2013-12-16 18:33:10,322][DEBUG][discovery.zen.elect ] [Batragon]
using minimum_master_nodes [1]

[2013-12-16 18:33:10,324][DEBUG][discovery.zen.fd ] [Batragon]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2013-12-16 18:33:10,328][DEBUG][discovery.zen.fd ] [Batragon]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2013-12-16 18:33:11,617][INFO ][node ] [Batragon]
initialized

[2013-12-16 18:33:11,617][INFO ][node ] [Batragon]
starting ..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f0c8749f-242a-4700-8575-db0c77264708%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Hi James,

Good to hear you found a work around. Still, I'd recommend finding out why
your settings are picked up as you'll probably run into this again in the
future. It may also be a problem in ES which we need to solve. If you can
pin it down, please open ticket by all means.

Cheers,
Boaz

On Mon, Dec 16, 2013 at 7:37 PM, James Beal zz9pzza@gmail.com wrote:

I can confirm that setting es.discovery.zen.minimum_master_nodes on the
command line looks as if it works unlike the setting in the config file.
Thank you Brian that looks like a workable work around.

I have additionally downloaded the deb file and the binaries and libs are
identical to my repackaged version

I have added the following to my defaults section and this looks like it
works.

Grab the number of hosts

HOSTS=grep discovery.zen.ping.unicast.hosts: $CONF_FILE | grep -v "^#" | awk -F: '{print $2}' | sed -e 's/\[//' -e 's/\]//'

NHOSTS=$(echo $(echo $HOSTS | sed "s/[,]/ /g" | wc -w))

MIN_MASTERS=$((($NHOSTS / 2) + 1))

Additional Java OPTS

ES_JAVA_OPTS="-Des.max-open-files=true
-Des.discovery.zen.minimum_master_nodes=$MIN_MASTERS"

And the log.

[2013-12-16 18:33:08,480][INFO ][bootstrap ] max_open_files
[99967]

[2013-12-16 18:33:08,617][INFO ][node ] [Batragon]
version[0.90.7], pid[1450], build[36897d0/2013-11-13T12:06:54Z]

[2013-12-16 18:33:08,617][INFO ][node ] [Batragon]
initializing ...

[2013-12-16 18:33:08,624][INFO ][plugins ] [Batragon]
loaded , sites

[2013-12-16 18:33:10,316][DEBUG][discovery.zen.ping.multicast] [Batragon]
using group [224.2.2.4], with port [54328], ttl [3], and address [null]

[2013-12-16 18:33:10,320][DEBUG][discovery.zen.ping.unicast] [Batragon]
using initial hosts , with concurrent_connects [10]

[2013-12-16 18:33:10,321][DEBUG][discovery.zen ] [Batragon]
using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]

[2013-12-16 18:33:10,322][DEBUG][discovery.zen.elect ] [Batragon]
using minimum_master_nodes [1]

[2013-12-16 18:33:10,324][DEBUG][discovery.zen.fd ] [Batragon]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2013-12-16 18:33:10,328][DEBUG][discovery.zen.fd ] [Batragon]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2013-12-16 18:33:11,617][INFO ][node ] [Batragon]
initialized

[2013-12-16 18:33:11,617][INFO ][node ] [Batragon]
starting ..

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vye1pNd5onw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0c8749f-242a-4700-8575-db0c77264708%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKzwz0rJrQQ0_ixkm5XOROsWHyTH4_iXcaH3kTN%2BQF4yHCO6SA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

On Tuesday, December 17, 2013 8:10:24 AM UTC, Boaz Leskes wrote:

Hi James,

Good to hear you found a work around. Still, I'd recommend finding out why
your settings are picked up as you'll probably run into this again in the
future. It may also be a problem in ES which we need to solve. If you can
pin it down, please open ticket by all means.

Cheers,
Boaz

I would love to work out why it is not working, Have you any ideas where I
would look ?

I could open a ticket with the information I and answer any questions
people have ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bf88888f-ad19-4927-9cdb-86ce65e369af%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

I think it would help a lot if you can reduce the issue to a short
reproduction which is independent of your setup. After that we can open an
issue if needed.

On Tue, Dec 17, 2013 at 9:23 AM, James Beal zz9pzza@gmail.com wrote:

On Tuesday, December 17, 2013 8:10:24 AM UTC, Boaz Leskes wrote:

Hi James,

Good to hear you found a work around. Still, I'd recommend finding out
why your settings are picked up as you'll probably run into this again in
the future. It may also be a problem in ES which we need to solve. If you
can pin it down, please open ticket by all means.

Cheers,
Boaz

I would love to work out why it is not working, Have you any ideas where I
would look ?

I could open a ticket with the information I and answer any questions
people have ?

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vye1pNd5onw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bf88888f-ad19-4927-9cdb-86ce65e369af%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKzwz0r8sL12UiByY9EaL8QFn0-bQZHHXke2CxoAGY4tHzomKw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

Hey James,

can you confirm, that there is no elasticsearch.yml in the direcrtory you
configured as es.default.path.conf property?

--Alex

On Mon, Dec 16, 2013 at 6:31 PM, James Beal zz9pzza@gmail.com wrote:

On Monday, December 16, 2013 12:30:39 PM UTC, Boaz Leskes wrote:

Hi James,

Everything looks good to me, so I'm puzzled by the ignored settings. A
couple of questions:

  1. Did you use a package to install ES?

No I pulled down the tar file and packaged it as I don't think originally
you did a deb file. I will have a look at the .deb file you have and see if
that would work for us.

  1. can you check if there is another elasticsearch.yml
    in /home/ao3elastic/conf with a different setting?

I can't see any other files...

root@test-elastic01:/home/james_# find /usr/otw /home/ao3elastic
/usr/local -name "elasticsearch_config.yml" -print

/home/ao3elastic/conf/elasticsearch_config.yml

  1. Are other settings from the elasticsearch_config.yml picked up? Did
    you change the cluster name?

Yes I changed the name and that took...

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "ao3elastic",

Boaz

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/50cafc60-1c6d-49ba-828d-ccc77c087c2e%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-qWFo5-1rUkRTxGyUjUuybHb823SyoojWss2Ez3NWr%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

James,

Just to check: The command line overrides begin with the es. prefix, but
that prefix must be omitted from the elasticsearch.yml file. In other words:

On the command line: -Des.discovery.zen.minimum_master_nodes=$MIN_MASTERS

In elasticsearch.yml: discovery.zen.minimum_master_nodes=3

I like the command line better, not as a work-around, but because it
automatically calculates the recommended value so that I don't need to do
it manually.

Also, I find the following command-line settings useful. Especially the
second one, so that from the command line I never guess where the config
file is: I know exactly where it is, and I know that ES will use that exact
file, and there's one less thing to worry about:

Set the node name to the hostname (without the domain, if present)

NODE_OPT="-Des.node.name=$(uname -n | cut -d'.' -f1)"

Store the Elasticsearch configuration outside of the ES_HOME directory

so its settings are preserved during reinstallations and upgrades

CONFIG_OPT="-Des.path.conf=/opt/cluster/config"

Regards,
Brian

On Tuesday, December 17, 2013 3:23:27 AM UTC-5, James Beal wrote:

I would love to work out why it is not working, Have you any ideas where I
would look ?

I could open a ticket with the information I and answer any questions
people have ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7cfad9a9-a720-47dd-8b96-cc1a680742b4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

15

On Tuesday, December 17, 2013 3:05:52 PM UTC, Alexander Reelsen wrote:

Hey James,

can you confirm, that there is no elasticsearch.yml in the direcrtory you
configured as es.default.path.conf property?

I can confirm that the is no elasticsearch.yml in that directory.

root@test-elastic01:/# ps -ef |grep java

6008 1450 1 0 Dec16 ? 00:06:35
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms36864m -Xmx36864m -Xmn512m
-Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Des.max-open-files=true
-Des.discovery.zen.minimum_master_nodes=1 -Delasticsearch
-Des.pidfile=/var/run/ao3elastic.pid
-Des.path.home=/usr/local/elasticsearch -cp
:/usr/local/elasticsearch/lib/elasticsearch-0.90.7.jar:/usr/local/elasticsearch/lib/:/usr/local/elasticsearch/lib/sigar/
-Des.node.data=true -Des.node.master=true
-Des.default.config=/home/ao3elastic/conf/elasticsearch_config.yml
-Des.default.path.home=/usr/local/elasticsearch/
-Des.default.path.logs=/home/ao3elastic/log
-Des.default.path.data=/home/ao3elastic/data
-Des.default.path.work=/home/ao3elastic/tmp
-Des.default.path.conf=/home/ao3elastic/conf
org.elasticsearch.bootstrap.Elasticsearch

root 27089 20119 0 20:06 pts/0 00:00:00 grep java

root@test-elastic01:/# ps -ef |grep j^C

root@test-elastic01:/# cd /home/ao3elastic/conf

root@test-elastic01:/home/ao3elastic/conf# ls -l

total 24

-rw-r----- 1 ao3elastic elastic 13041 Dec 15 23:07 elasticsearch_config.yml

-rw-r----- 1 ao3elastic elastic 1220 Dec 17 19:27 elasticsearch_defaults

-rw-r----- 1 ao3elastic elastic 1373 Dec 11 21:49 logging.yml

root@test-elastic01:/home/ao3elastic/conf#

There is one in /usr/otw/elasticsearch-0.90.7/config/elasticsearch.yml

which is a link to /usr/local/elasticsearch/config/elasticsearch.yml

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8a0ec7cc-9684-4c17-bac0-d7de84e68260%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

16

On Tuesday, December 17, 2013 6:53:35 PM UTC, InquiringMind wrote:

James,

Just to check: The command line overrides begin with the es. prefix, but
that prefix must be omitted from the elasticsearch.yml file. In other words:

On the command line: -Des.discovery.zen.minimum_master_nodes=$MIN_MASTERS

In elasticsearch.yml: discovery.zen.minimum_master_nodes=3

I have the following, note the : rather than the = however that is
following the other config options.

discovery.zen.minimum_master_nodes: 1

I like the command line better, not as a work-around, but because it
automatically calculates the recommended value so that I don't need to do
it manually.

Also, I find the following command-line settings useful. Especially the
second one, so that from the command line I never guess where the config
file is: I know exactly where it is, and I know that ES will use that exact
file, and there's one less thing to worry about:

Set the node name to the hostname (without the domain, if present)

NODE_OPT="-Des.node.name=$(uname -n | cut -d'.' -f1)"

I can see the sense in that ( although it is functional rather than fun... )

Store the Elasticsearch configuration outside of the ES_HOME directory

so its settings are preserved during reinstallations and upgrades

CONFIG_OPT="-Des.path.conf=/opt/cluster/config"

This I do already :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2ed2d4d9-1025-4d4f-8104-71949456b3d1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

17

James,

Good catch (concerning the = sign vrs. a colon). I saw it and wished I
could edit my post.

Regards,
Brian

On Tuesday, December 17, 2013 5:03:43 PM UTC-5, James Beal wrote:

On Tuesday, December 17, 2013 6:53:35 PM UTC, InquiringMind wrote:

James,

Just to check: The command line overrides begin with the es. prefix, but
that prefix must be omitted from the elasticsearch.yml file. In other words:

On the command line: -Des.discovery.zen.minimum_master_nodes=$MIN_MASTERS

In elasticsearch.yml: discovery.zen.minimum_master_nodes=3

I have the following, note the : rather than the = however that is
following the other config options.

discovery.zen.minimum_master_nodes: 1

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e1cd9aad-a25c-4c10-9277-38eaf10f9443%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.