Greetings folks,
I have a cluster with two nodes (node1 and node2) running ELK 7.12, I activated security feature from this link Set up minimal security for Elasticsearch | Elasticsearch Guide [7.12] | Elastic, I followed all the steps, I successfully log in to kibana with a user/password, but unfortunately my cluster state is yellow, unable to reach my second node.
I run the command ./bin/elasticsearch-setup-passwords interactive from node1, when I try to run the same command from node2, I get the following error:
Checking cluster health: http://10.0.1.5:9200/_cluster/health?pretty
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
Failed to determine the health of the cluster running at http://10.0.1.5:9200
Unexpected response code [503] from calling GET http://10.0.1.5:9200/_cluster/health?pretty
Cause: master_not_discovered_exception
It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.
Do you want to continue with the password setup process [y/N]
The result command of cluster health:
{
"cluster_name" : "my-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 14,
"active_shards" : 14,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 77.77777777777779
}
elasticsearch.yml for node1:
cluster.name: my-cluster
cluster.initial_master_nodes: ["node1", "node2"]
node.name: node1
# Path to log files:
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.seed_hosts: ["node1", "node2"]
node.master: true
node.data: true
node.ingest: true
node.ml: false
node.transform: false
node.remote_cluster_client: false
## xpack
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
elasticsearch.yml for node2:
cluster.name: my-cluster
node.name: node2
network.host: 0.0.0.0
discovery.seed_hosts: ["node1", "node2"]
node.master: true
node.data: true
node.ingest: true
node.ml: false
node.transform: false
node.remote_cluster_client: false
## xpack
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
elasticsearch log from node1:
[2021-04-15T21:32:47,389][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45270, profile=default}
[2021-04-15T21:32:48,389][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45274, profile=default}
[2021-04-15T21:32:49,389][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45278, profile=default}
[2021-04-15T21:32:50,390][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45282, profile=default}
[2021-04-15T21:32:51,393][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45284, profile=default}
[2021-04-15T21:32:52,390][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45286, profile=default}
[2021-04-15T21:32:53,390][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45296, profile=default}
[2021-04-15T21:32:54,390][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.1.4:9300, remoteAddress=/10.0.1.5:45298, profile=default}
elasticsearch log from node2:
[2021-04-15T21:32:58,325][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered or elected yet, an election requires one or more nodes that have already participated as master-eligible nodes in the cluster but this node was not master-eligible the last time it joined the cluster, have discovered [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] which is not a quorum; discovery will continue using [10.0.1.4:9300, 127.0.0.1:9300] from hosts providers and [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] from last-known cluster state; node term 26, last-accepted version 769 in term 26
[2021-04-15T21:33:07,251][WARN ][r.suppressed ] [node2] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:219) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:324) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:241) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:590) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.12.0.jar:7.12.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-04-15T21:33:08,326][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered or elected yet, an election requires one or more nodes that have already participated as master-eligible nodes in the cluster but this node was not master-eligible the last time it joined the cluster, have discovered [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] which is not a quorum; discovery will continue using [10.0.1.4:9300, 127.0.0.1:9300] from hosts providers and [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] from last-known cluster state; node term 26, last-accepted version 769 in term 26[2021-04-15T21:32:58,325][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered or elected yet, an election requires one or more nodes that have already participated as master-eligible nodes in the cluster but this node was not master-eligible the last time it joined the cluster, have discovered [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] which is not a quorum; discovery will continue using [10.0.1.4:9300, 127.0.0.1:9300] from hosts providers and [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] from last-known cluster state; node term 26, last-accepted version 769 in term 26
[2021-04-15T21:33:07,251][WARN ][r.suppressed ] [node2] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:219) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:324) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:241) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:590) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.12.0.jar:7.12.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-04-15T21:33:08,326][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered or elected yet, an election requires one or more nodes that have already participated as master-eligible nodes in the cluster but this node was not master-eligible the last time it joined the cluster, have discovered [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] which is not a quorum; discovery will continue using [10.0.1.4:9300, 127.0.0.1:9300] from hosts providers and [{node2}{8IbR4qo-QS-bYp5FuLbEtw}{qk42rRncT8yQVWHrBdHyzg}{10.0.1.5}{10.0.1.5:9300}{cdfhimsw}{xpack.installed=true, transform.node=false}] from last-known cluster state; node term 26, last-accepted version 769 in term 26
What I am missing please ?
, indeed insufficient permission for the