I am trying to deploy a 2 node(1 master,1 data) elasticsearch cluster using docker and was able to start the first node as master without any issues but when I try to start the second elasticsearch docker container it can't discover the master. Both machines are in the same network, able to ping to the master es port from data machine terminal.
Elastic search version : 5.6.2
Docker version : 1.13.1
Getting this startup log for data node :
[2019-10-14T10:50:44,194][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-10-14T10:50:44,196][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2019-10-14T10:50:44,196][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-10-14T10:50:44,197][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2019-10-14T10:50:44,197][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-10-14T10:50:44,311][INFO ][o.e.n.Node ] [data-1] initializing ...
[2019-10-14T10:50:44,392][INFO ][o.e.e.NodeEnvironment ] [data-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [7.4gb], net total_space [9.6gb], spins? [unknown], types [rootfs]
[2019-10-14T10:50:44,393][INFO ][o.e.e.NodeEnvironment ] [data-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2019-10-14T10:50:44,394][INFO ][o.e.n.Node ] [data-1] node name [data-1], node ID [3LHyDPPtRrWQz8-CaSMztA]
[2019-10-14T10:50:44,394][INFO ][o.e.n.Node ] [data-1] version[5.6.2], pid[1], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/3.10.0-957.21.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
[2019-10-14T10:50:44,395][INFO ][o.e.n.Node ] [data-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
[2019-10-14T10:50:46,103][INFO ][o.e.p.PluginsService ] [data-1] loaded module [aggs-matrix-stats]
[2019-10-14T10:50:46,103][INFO ][o.e.p.PluginsService ] [data-1] loaded module [ingest-common]
[2019-10-14T10:50:46,103][INFO ][o.e.p.PluginsService ] [data-1] loaded module [lang-expression]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [lang-groovy]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [lang-mustache]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [lang-painless]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [parent-join]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [percolator]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [reindex]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [transport-netty3]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded module [transport-netty4]
[2019-10-14T10:50:46,104][INFO ][o.e.p.PluginsService ] [data-1] loaded plugin [ingest-geoip]
[2019-10-14T10:50:46,105][INFO ][o.e.p.PluginsService ] [data-1] loaded plugin [ingest-user-agent]
[2019-10-14T10:50:46,105][INFO ][o.e.p.PluginsService ] [data-1] loaded plugin [x-pack]
[2019-10-14T10:50:48,321][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/133] [Main.cc@128] controller (64 bit): Version 5.6.2 (Build 228329870d1c63) Copyright (c) 2017 Elasticsearch BV
[2019-10-14T10:50:48,365][INFO ][o.e.d.DiscoveryModule ] [data-1] using discovery type [zen]
[2019-10-14T10:50:49,227][INFO ][o.e.n.Node ] [data-1] initialized
[2019-10-14T10:50:49,227][INFO ][o.e.n.Node ] [data-1] starting ...
[2019-10-14T10:50:49,517][INFO ][o.e.t.TransportService ] [data-1] publish_address {127.0.0.1:9301}, bound_addresses {127.0.0.1:9301}
[2019-10-14T10:50:49,529][WARN ][o.e.b.BootstrapChecks ] [data-1] memory locking requested for elasticsearch process but memory is not locked
[2019-10-14T10:50:52,566][WARN ][o.e.d.z.ZenDiscovery ] [data-1] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[2019-10-14T10:50:55,568][WARN ][o.e.d.z.ZenDiscovery ] [data-1] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[2019-10-14T10:50:58,570][WARN ][o.e.d.z.ZenDiscovery ] [data-1] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[2019-10-14T10:51:19,554][WARN ][o.e.n.Node ] [data-1] timed out while waiting for initial discovery state - timeout: 30s
[2019-10-14T10:51:19,571][INFO ][o.e.h.n.Netty4HttpServerTransport] [data-1] publish_address {172.20.24.99:9200}, bound_addresses {172.21.0.2:9200}
[2019-10-14T10:51:19,572][INFO ][o.e.n.Node ] [data-1] started
[2019-10-14T10:51:19,585][WARN ][o.e.d.z.ZenDiscovery ] [data-1] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[2019-10-14T10:51:22,586][WARN ][o.e.d.z.ZenDiscovery ] [data-1] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[2019-10-14T10:51:25,588][WARN ][o.e.d.z.ZenDiscovery ] [data-1] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
Data node netsats -lnp
Master Node config-file
cluster.name: mynewclustername
node.name: master
bootstrap.memory_lock: true
network.host: site
http.port: 9200
network.publish_host: 172.20.7.29
discovery.zen.ping.unicast.hosts: ["172.20.7.29"]
xpack.security.enabled: false
node.master: true
node.data: false
Data Node config-file
cluster.name: mynewclustername
node.name: data-1
bootstrap.memory_lock: true
network.host: site
http.port: 9200
network.publish_host: 172.20.24.99
transport.host: localhost
discovery.zen.ping.unicast.hosts: ["172.20.7.29"]
xpack.security.enabled: false
node.master: false
node.data: true