Elasticsearch cluster - 7.6 Error start

Hi, first sorry my english not be good, but i have a doubt, i am creating a cluster of elasticsearch,where I have 3 nodes,
but there is an error in the climb

Conf Master node:
cluster.name: elk_lab
node.master: true
node.data: false
node.name: node-3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#path.data: /opt/elk/data
network.host: IP1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["IP1", "IP2", "IP3"]
discovery.zen.minimum_master_nodes: 1

conf Node Data 1:
cluster.name: elk_lab
node.master: false
node.data: true
node.name: node-2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: IP2
http.port: 9200
discovery.zen.ping.unicast.hosts: ["IP1", "IP2", "IP3"]
discovery.zen.minimum_master_nodes: 1
cluster.initial_master_nodes: ["IP1"]

conf Node Data 2:
cluster.name: elk_lab
node.master: false
node.data: true
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: IP3
http.port: 9200
discovery.zen.ping.unicast.hosts: ["IP1", "IP2", "IP3"]
discovery.zen.minimum_master_nodes: 1
cluster.initial_master_nodes: ["IP1"]

Logs Master:
[2020-02-26T23:17:41,008][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{k1BO077zSLm9Iqff1nLqMg}{tEK5LrK9SE2E-s_nO4D3aQ}{192.168.99.10}{192.168.99.10:9300}{ilm}{ml.machine_memory=3897933824, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using

Log Node 1
[2020-02-27T02:17:07,339][WARN ][r.suppressed ] [node-2] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:535) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:415) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:569) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Suppressed: org.elasticsearch.discovery.MasterNotDiscoveredException
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-02-27T02:17:12,389][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [node-2] no known master node, scheduling a retry
[2020-02-27T02:17:14,960][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-2] master not discovered yet: have discovered [{node-2}{3YbiaxxFQhSyv066zU4-nw}{daDu-6gmQzG0MPRaBAZCgA}{192.168.99.40}{192.168.99.40:9300}{dil}{ml.machine_memory=2933784576, xpack.installed=true, ml.max_open_jobs=20}, {node-3}{k1BO077zSLm9Iqff1nLqMg}{tEK5LrK9SE2E-s_nO4D3aQ}{192.168.99.10}{192.168.99.10:9300}{ilm}{ml.machine_memory=3897933824, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [192.168.99.10:9300, 192.168.99.50:9300] from hosts providers and from last-known cluster state; node term 22, last-accepted version 799 in term 22

Log Node 2
[2020-02-27T03:34:03,162][WARN ][r.suppressed ] [node-1] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:535) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:415) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:569) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Suppressed: org.elasticsearch.discovery.MasterNotDiscoveredException
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-02-27T03:34:08,779][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet: have discovered [{node-1}{LHO5p-BKRlmnl4Wfa8Ar7g}{Sub_jW1xSIe2VgYyUVcfaA}{192.168.99.50}{192.168.99.50:9300}{dil}{ml.machine_memory=2933948416, xpack.installed=true, ml.max_open_jobs=20}, {node-3}{k1BO077zSLm9Iqff1nLqMg}{tEK5LrK9SE2E-s_nO4D3aQ}{192.168.99.10}{192.168.99.10:9300}{ilm}{ml.machine_memory=3897933824, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [192.168.99.10:9300, 192.168.99.40:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0

Log Logstash:
[2020-02-26T23:18:51,096][WARN ][logstash.outputs.elasticsearch][Chamados] DEPRECATION WARNING: Connecting to an OSS distribution of Elasticsearch using the default distribution of Logstash will stop working in Logstash 8.0.0. Please upgrade to the default distribution of Elasticsearch, or use the OSS distribution of Logstash {:url=>"http://192.168.99.50:9200/"}
[2020-02-26T23:18:51,105][WARN ][logstash.outputs.elasticsearch][Chamados] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://192.168.99.40:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://192.168.99.40:9200/, :error_message=>"Elasticsearch Unreachable: [http://192.168.99.40:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-02-26T23:18:51,116][ERROR][logstash.outputs.elasticsearch][Chamados] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://192.168.99.40:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-02-26T23:18:52,131][WARN ][logstash.outputs.elasticsearch][Chamados] Restored connection to ES instance {:url=>"http://192.168.99.40:9200/"}
[2020-02-26T23:19:00,331][INFO ][logstash.inputs.jdbc ][Chamados] (0.003611s) select * from Chamados

Broadcast message from root@elk (Wed 2020-02-26 23:19:05 -03):

The system is going down for power-off at Wed 2020-02-26 23:20:05 -03!

[2020-02-26T23:19:22,221][WARN ][logstash.outputs.elasticsearch][Chamados] DEPRECATION WARNING: Connecting to an OSS distribution of Elasticsearch using the default distribution of Logstash will stop working in Logstash 8.0.0. Please upgrade to the default distribution of Elasticsearch, or use the OSS distribution of Logstash {:url=>"http://192.168.99.40:9200/"}
[2020-02-26T23:20:00,380][INFO ][logstash.inputs.jdbc ][Chamados] (0.003266s) select * from Chamados

Log Kibana:

{"type":"log","@timestamp":"2020-02-27T02:13:41Z","tags":["warning","plugins","licensing"],"pid":2611,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. TypeError: Cannot use 'in' operator to search for 'type' in null"}
{"type":"log","@timestamp":"2020-02-27T02:14:11Z","tags":["warning","plugins","licensing"],"pid":2611,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. TypeError: Cannot use 'in' operator to search for 'type' in null"}

http://192.168.99.40:9200/_cat/health

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

http://192.168.99.10:9200/_cat/indices
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

http://192.168.99.50:9200/_cat/health
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

The master logs are giving you a big hint:

You have set cluster.initial_master_nodes on the two non-master nodes (where it is ignored) but not on the master node (where it is required the first time the cluster starts).

Also you should set it to the node.name property not an IP address:

cluster.initial_master_nodes: ["node-3"]

Very thanks @DavidTurner, but i add the config [cluster.initial_master_nodes]
in the nodes too, would have a problem? again very thanks

Sorry, I don't really understand what you're asking here. Does the documentation help at all?

@DavidTurner, the first solution solved the problem

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.