I am attempting to stand up 3 master nodes, 2 client nodes 3 data nodes and also Logstash and Kibana in my setup of ELK in AWS. I am starting with the master nodes for the cluster. Here is the elasticsearch.yml config file from all 3 master nodes:
`cluster.name: ELK-DEV
cloud.aws.access_key: removed
cloud.aws.secret_key: removed
node.name: ${HOSTNAME}
node.master: true
node.data: false
discovery.type: ec2
discovery.ec2.node_cache_time: 120s
discovery.ec2.tag.es_cluster: ELK - DEV
discovery.zen.ping_timeout: 30s
discovery.ec2.groups: EC2 - DEV
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts:
- datanode1 ip4 address
- datanode 2 ip4 address
- datanode 3 ip4 address
- clientnode 1 ip4 address
- clientnode 2 ip4 address
- masternode 2 ip4 address
- masternode 3 ip4 address
discovery.zen.hosts_provider: ec2
discovery.zen.join_timeout: 90s
cloud.aws.region: us-east-1
cloud.aws.protocol: https
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.availability_zones: us-east-1a,us-east-1b
path.data: "/etc/elasticsearch/data"
path.logs: "/var/log/elasticsearch"
bootstrap.memory_lock: true
http.port: 9200
transport.tcp.port: 9300
plugin.mandatory: - discovery-ec2
network.host: - eth0:ipv4
-
local
network.bind_host: eth0:ipv4
network.publish_host: 10.0.0.65`
When I run curl -XGET 'ec2-public-ip-amazonaws.com:9200/_cluster/health?pretty' I get the following error message:
{ "error" : { "root_cause" : [ { "type" : "master_not_discovered_exception", "reason" : null } ], "type" : "master_not_discovered_exception", "reason" : null }, "status" : 503 }
This is what I am seeing in the log file:
[2017-04-10T19:20:48,944][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [elk-dev-master1] no known master node, scheduling a retry [2017-04-10T19:21:18,946][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [elk-dev-master1] timed out while retrying [cluster:monitor/health] after failure (timeout [30s]) [2017-04-10T19:21:18,947][WARN ][r.suppressed ] path: /_cluster/health, params: {pretty=} org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:209) [elasticsearch-5.3.0.jar:5.3.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:311) [elasticsearch-5.3.0.jar:5.3.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:238) [elasticsearch-5.3.0.jar:5.3.0] at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:1162) [elasticsearch-5.3.0.jar:5.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:544) [elasticsearch-5.3.0.jar:5.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
When I run a netstat -tuplen here are the results:
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 9387 900/sshd tcp6 0 0 10.0.0.65:9200 :::* LISTEN 106 76095 8839/java tcp6 0 0 10.0.0.65:9300 :::* LISTEN 106 76079 8839/java tcp6 0 0 :::22 :::* LISTEN 0 9389 900/sshd udp 0 0 0.0.0.0:3447 0.0.0.0:* 0 7903 578/dhclient udp 0 0 0.0.0.0:68 0.0.0.0:* 0 1822 578/dhclient udp6 0 0 :::11290 :::* 0 7904 578/dhclient
I've tried restarting the elasticsearch services. Any help would be great. Thank in advance.