I am setting up a 7 node cluster with 3 Masters / 3 Data / 1 Coordinating node
The cluster is deployed on AWS with ec2 discovery plugin.
The nodes are using Amazon Linux 2 and the ES/Kibana versions are 6.4.2
kibana.yml:
server.port: 8080
server.host: 10.11.11.11 # private ip of node
elasticsearch.url: "http://localhost:9200"
/etc/elasticsearch/elasticsearch.yml
cluster.name: elastic-cluster-prod
path.data: /elastic_data
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
discovery.ec2.host_type: private_ip
discovery.zen.hosts_provider: ec2
http.port: 9200
discovery.zen.minimum_master_nodes: 2
node.master: false
node.data: false
node.ingest: false
s3.client.default.endpoint: s3-eu-west-1.amazonaws.com
transport.tcp.port: 9300
node.name: es-prod-coord
action.auto_create_index: true
transport.host: _ec2:privateIp_
However, although all other nodes seem healthy, coordinating node fails:
kibana.log
{"type":"error","@timestamp":"2019-01-03T08:18:17Z","tags":["warning","stats-collection"],"pid":2907,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.4.2]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.4.2]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.4.2\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.4.2]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.4.2]: routing [null]]\\\"},\\\"status\\\":503}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickDomainCallback (internal/process/next_tick.js:218:9)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.4.2]: routing [null]]"}
{"type":"log","@timestamp":"2019-01-03T08:18:17Z","tags":["warning","stats-collection"],"pid":2907,"message":"Unable to fetch data from kibana_settings collector"}
/var/log/elasticsearch/<cluster-name>.log
[2019-01-03T08:19:17,969][WARN ][r.suppressed ] path: /.kibana/doc/config%3A6.4.2, params: {index=.kibana, id=config:6.4.2, type=doc}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.4.2]: routing [null]]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:207) ~[elasticsearch-6.4.2.jar:6.4.2]
at o [...]