Unable to get Docker ELK cluster working with 2 hosts. Both hosts have same response:
user01@dcw0970:~/docker-elk-master$ docker logs dockerelkmaster_elasticsearch_1
[2018-02-15T01:33:13,054][INFO ][o.e.n.Node ] [896e8523da84] initializing ...
[2018-02-15T01:33:13,123][INFO ][o.e.e.NodeEnvironment ] [896e8523da84] using [1] data paths, mounts [[/ (none)]], net usable_space [5.2gb], net total_space [7.3gb], types [aufs]
[2018-02-15T01:33:13,123][INFO ][o.e.e.NodeEnvironment ] [896e8523da84] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-02-15T01:33:13,125][INFO ][o.e.n.Node ] [896e8523da84] node name [896e8523da84], node ID [lXvb0g0PT1qKX9i7h8DvcA]
[2018-02-15T01:33:13,125][INFO ][o.e.n.Node ] [896e8523da84] version[6.0.0], pid[1], build[8f0685b/2017-11-10T18:41:22.859Z], OS[Linux/4.4.0-98-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-02-15T01:33:13,125][INFO ][o.e.n.Node ] [896e8523da84] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xmx1g, -Xms1g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [aggs-matrix-stats]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [analysis-common]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [ingest-common]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [lang-expression]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [lang-mustache]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [lang-painless]
[2018-02-15T01:33:13,839][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [parent-join]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [percolator]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [reindex]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [repository-url]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [transport-netty4]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded module [tribe]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded plugin [ingest-geoip]
[2018-02-15T01:33:13,840][INFO ][o.e.p.PluginsService ] [896e8523da84] loaded plugin [ingest-user-agent]
[2018-02-15T01:33:14,877][INFO ][o.e.d.DiscoveryModule ] [896e8523da84] using discovery type [zen]
[2018-02-15T01:33:15,331][INFO ][o.e.n.Node ] [896e8523da84] initialized
[2018-02-15T01:33:15,331][INFO ][o.e.n.Node ] [896e8523da84] starting ...
[2018-02-15T01:33:15,537][INFO ][o.e.t.TransportService ] [896e8523da84] publish_address {xx.xxx.xx.248:9300}, bound_addresses {0.0.0.0:9300}
[2018-02-15T01:33:15,547][INFO ][o.e.b.BootstrapChecks ] [896e8523da84] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-02-15T01:33:15,622][WARN ][o.e.t.n.Netty4Transport ] [896e8523da84] write and flush on the network layer failed (channel: [id: 0xace1b77f, L:0.0.0.0/0.0.0.0:59142 ! R:dcw0970.dev.elk.com/xx.xxx.xx.248:9200])
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-02-15T01:33:16,576][WARN ][o.e.t.n.Netty4Transport ] [896e8523da84] write and flush on the network layer failed (channel: [id: 0xa1107c7c, L:0.0.0.0/0.0.0.0:59148 ! R:dcw0970.dev.elk.com/xx.xxx.xx.248:9200])
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-02-15T01:33:17,577][WARN ][o.e.t.n.Netty4Transport ] [896e8523da84] write and flush on the network layer failed (channel: [id: 0xb3e960d8, L:0.0.0.0/0.0.0.0:59156 ! R:dcw0970.dev.elk.com/xx.xxx.xx.248:9200])
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-02-15T01:33:18,596][INFO ][o.e.c.s.MasterService ] [896e8523da84] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {896e8523da84}{lXvb0g0PT1qKX9i7h8DvcA}{i1QPfGm0SFi2kueAG9NBqw}{xx.xxx.xx.248}{xx.xxx.xx.248:9300}
[2018-02-15T01:33:18,608][INFO ][o.e.c.s.ClusterApplierService] [896e8523da84] new_master {896e8523da84}{lXvb0g0PT1qKX9i7h8DvcA}{i1QPfGm0SFi2kueAG9NBqw}{xx.xxx.xx.248}{xx.xxx.xx.248:9300}, reason: apply cluster state (from master [master {896e8523da84}{lXvb0g0PT1qKX9i7h8DvcA}{i1QPfGm0SFi2kueAG9NBqw}{xx.xxx.xx.248}{xx.xxx.xx.248:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-02-15T01:33:18,621][INFO ][o.e.h.n.Netty4HttpServerTransport] [896e8523da84] publish_address {xx.xxx.xx.248:9200}, bound_addresses {0.0.0.0:9200}
[2018-02-15T01:33:18,621][INFO ][o.e.n.Node ] [896e8523da84] started
[2018-02-15T01:33:18,628][INFO ][o.e.g.GatewayService ] [896e8523da84] recovered [0] indices into cluster_state
elasticsearch.yml:
node.name: ${HOSTNAME}
cluster.name: "docker-dev-elk"
network.host: 0.0.0.0
network.publish_host: xx.xxx.xx.248
discovery.zen.minimum_master_nodes: 2
gateway.expected_nodes: 2
gateway.recover_after_time: 5m
gateway.recover_after_nodes: 2
discovery.zen.ping.unicast.hosts:
-dcw0970.dev.elk.com:9200
-dcw0971.dev.elk.com:9200
Both hosts have green status:
{
"cluster_name": "docker-dev-elk",
"status": "green",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}