Bootstrap issues

Using ES 5.4.2. Trying to setup a production grade cluster with 2 masters to start.

Here is my config....

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: mdm_dev_es
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: mdm_dev_esm_1
node.data: false
node.master: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 10.127.79.68
network.host: _site_
#
# Set a custom port for HTTP:
#
http.port: 9200
transport.tcp.port: 9300
#transport.host: 10.127.79.68
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts:
    - 10.127.78.116:9300
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#                                     

I see the following issues in my logs.

version[5.4.2], pid[23945], build[929b078/2017-06-15T02:29:28.122Z], OS[Linux/3.10.0-514.21.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b12]
[2017-06-22T19:59:11,511][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-06-22T19:59:12,252][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [aggs-matrix-stats]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [ingest-common]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-expression]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-groovy]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-mustache]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-painless]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [percolator]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [reindex]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [transport-netty3]
[2017-06-22T19:59:12,253][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [transport-netty4]
[2017-06-22T19:59:12,254][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] no plugins loaded
[2017-06-22T19:59:13,347][INFO ][o.e.d.DiscoveryModule    ] [mdm_dev_esm_1] using discovery type [zen]
[2017-06-22T19:59:13,758][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] initialized
[2017-06-22T19:59:13,758][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] starting ...
[2017-06-22T19:59:13,879][INFO ][o.e.t.TransportService   ] [mdm_dev_esm_1] publish_address {10.127.79.68:9300}, bound_addresses {10.127.79.68:9300}
[2017-06-22T19:59:13,885][INFO ][o.e.b.BootstrapChecks    ] [mdm_dev_esm_1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-06-22T19:59:13,886][ERROR][o.e.b.Bootstrap          ] [mdm_dev_esm_1] node validation exception
[1] bootstrap checks failed

What is the correct basic configuration for a working 2 node master cluster.

Everything you need to check is right after those lines:

[2017-06-22T19:59:13,886][ERROR][o.e.b.Bootstrap          ] [mdm_dev_esm_1] node validation exception
[1] bootstrap checks failed

Please read https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html

Hi David thanks for your response. Before posting the question I had tried a few transport options already and anything other than localhost gives me the error I posted above.I tried bypassing bootstrap checks as well before I posted and I get an error stating that the jvm option needs to be set to true when in fact I had set it to false. Hence in a production environment I wanted to know which combination of transport options and jvm options if any I need to set.

Paste all the logs you have and I might be able to help.

Hi David you can see my logs below with the following basic configuration that I currently have.
Please also see my comments in the config. Also notice the line publish_address {10.127.79.68:9300}, bound_addresses {10.127.79.68:9300} in the logs.

Due to character limit I will paste another set of logs with the JVM option.

cluster.name: mdm_dev_es
node.name: mdm_dev_esm_1
node.data: false
node.master: true
bootstrap.memory_lock: true
network.host: _site_  # tried with network.host: 0.0.0.0 with the same effect
http.port: 9200
transport.tcp.port: 9300
transport.host: 10.127.79.68
discovery.zen.ping.unicast.hosts:
    - 10.127.78.116:9300
discovery.zen.minimum_master_nodes: 2
service elasticsearch restart
Restarting elasticsearch (via systemctl):                  [  OK  ]
[root@ip-10-127-79-68 centos]# tail -f /var/log/elasticsearch/mdm_dev_es.log 
[2017-06-23T16:50:24,340][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] closing ...
[2017-06-23T16:50:24,347][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] closed
[2017-06-23T16:50:25,750][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2017-06-23T16:50:25,751][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-06-23T16:50:25,752][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-06-23T16:50:25,752][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2017-06-23T16:50:25,752][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-06-23T16:50:25,849][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] initializing ...
[2017-06-23T16:50:25,908][INFO ][o.e.e.NodeEnvironment    ] [mdm_dev_esm_1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [97.7gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2017-06-23T16:50:25,908][INFO ][o.e.e.NodeEnvironment    ] [mdm_dev_esm_1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-06-23T16:50:25,909][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] node name [mdm_dev_esm_1], node ID [hpDBqWmaTlOwcEaf3DlhxQ]
[2017-06-23T16:50:25,909][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] version[5.4.2], pid[32359], build[929b078/2017-06-15T02:29:28.122Z], OS[Linux/3.10.0-514.21.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b12]
[2017-06-23T16:50:25,910][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-06-23T16:50:26,650][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [aggs-matrix-stats]
[2017-06-23T16:50:26,650][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [ingest-common]
[2017-06-23T16:50:26,650][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-expression]
[2017-06-23T16:50:26,650][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-groovy]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-mustache]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [lang-painless]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [percolator]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [reindex]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [transport-netty3]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] loaded module [transport-netty4]
[2017-06-23T16:50:26,651][INFO ][o.e.p.PluginsService     ] [mdm_dev_esm_1] no plugins loaded
[2017-06-23T16:50:27,764][INFO ][o.e.d.DiscoveryModule    ] [mdm_dev_esm_1] using discovery type [zen]
[2017-06-23T16:50:28,187][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] initialized
[2017-06-23T16:50:28,187][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] starting ...
[2017-06-23T16:50:28,307][INFO ][o.e.t.TransportService   ] [mdm_dev_esm_1] publish_address {10.127.79.68:9300}, bound_addresses {10.127.79.68:9300}
[2017-06-23T16:50:28,312][INFO ][o.e.b.BootstrapChecks    ] [mdm_dev_esm_1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-06-23T16:50:28,314][ERROR][o.e.b.Bootstrap          ] [mdm_dev_esm_1] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2017-06-23T16:50:28,321][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] stopping ...
[2017-06-23T16:50:28,351][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] stopped
[2017-06-23T16:50:28,351][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] closing ...
[2017-06-23T16:50:28,361][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] closed

Continued from previous response...

When I added the JVM options -Des.enforce.bootstrap.checks=false I get the following error. See first line of the log below.

[2017-06-23T17:03:07,397][ERROR][o.e.b.Bootstrap          ] [mdm_dev_esm_1] Exception
java.lang.IllegalArgumentException: [es.enforce.bootstrap.checks] must be [true] but was [false]
	at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:127) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:95) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:76) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:237) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.node.Node.start(Node.java:687) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:277) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:359) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.2.jar:5.4.2]
[2017-06-23T17:03:07,408][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [mdm_dev_esm_1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: [es.enforce.bootstrap.checks] must be [true] but was [false]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.2.jar:5.4.2]
Caused by: java.lang.IllegalArgumentException: [es.enforce.bootstrap.checks] must be [true] but was [false]
	at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:127) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:95) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:76) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:237) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.node.Node.start(Node.java:687) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:277) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:359) ~[elasticsearch-5.4.2.jar:5.4.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.2.jar:5.4.2]
	... 6 more
[2017-06-23T17:03:07,414][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] stopping ...
[2017-06-23T17:03:07,431][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] stopped
[2017-06-23T17:03:07,431][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] closing ...
[2017-06-23T17:03:07,501][INFO ][o.e.n.Node               ] [mdm_dev_esm_1] closed

[1]: memory locking requested for elasticsearch process but memory is not locked

Did you follow the guide here? Memory lock check | Elasticsearch Guide [8.11] | Elastic

I saw that comment on the guide: Disable swapping | Elasticsearch Guide [8.11] | Elastic

mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available!

May be this is what is happening here?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.