Hi, first sorry my english not be good, but i have a doubt, i am creating a cluster of elasticsearch,where I have 3 nodes,
but there is an error in the climb
Conf Master node:
cluster.name: elk_lab
node.master: true
node.data: false
node.name: node-3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#path.data: /opt/elk/data
network.host: IP1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["IP1", "IP2", "IP3"]
discovery.zen.minimum_master_nodes: 1
conf Node Data 1:
cluster.name: elk_lab
node.master: false
node.data: true
node.name: node-2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: IP2
http.port: 9200
discovery.zen.ping.unicast.hosts: ["IP1", "IP2", "IP3"]
discovery.zen.minimum_master_nodes: 1
cluster.initial_master_nodes: ["IP1"]
conf Node Data 2:
cluster.name: elk_lab
node.master: false
node.data: true
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: IP3
http.port: 9200
discovery.zen.ping.unicast.hosts: ["IP1", "IP2", "IP3"]
discovery.zen.minimum_master_nodes: 1
cluster.initial_master_nodes: ["IP1"]
Logs Master:
[2020-02-26T23:17:41,008][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{k1BO077zSLm9Iqff1nLqMg}{tEK5LrK9SE2E-s_nO4D3aQ}{192.168.99.10}{192.168.99.10:9300}{ilm}{ml.machine_memory=3897933824, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using
Log Node 1
[2020-02-27T02:17:07,339][WARN ][r.suppressed ] [node-2] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:535) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:415) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:569) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Suppressed: org.elasticsearch.discovery.MasterNotDiscoveredException
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-02-27T02:17:12,389][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [node-2] no known master node, scheduling a retry
[2020-02-27T02:17:14,960][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-2] master not discovered yet: have discovered [{node-2}{3YbiaxxFQhSyv066zU4-nw}{daDu-6gmQzG0MPRaBAZCgA}{192.168.99.40}{192.168.99.40:9300}{dil}{ml.machine_memory=2933784576, xpack.installed=true, ml.max_open_jobs=20}, {node-3}{k1BO077zSLm9Iqff1nLqMg}{tEK5LrK9SE2E-s_nO4D3aQ}{192.168.99.10}{192.168.99.10:9300}{ilm}{ml.machine_memory=3897933824, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [192.168.99.10:9300, 192.168.99.50:9300] from hosts providers and from last-known cluster state; node term 22, last-accepted version 799 in term 22
Log Node 2
[2020-02-27T03:34:03,162][WARN ][r.suppressed ] [node-1] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:535) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:415) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:569) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Suppressed: org.elasticsearch.discovery.MasterNotDiscoveredException
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) ~[elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.0.jar:7.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-02-27T03:34:08,779][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet: have discovered [{node-1}{LHO5p-BKRlmnl4Wfa8Ar7g}{Sub_jW1xSIe2VgYyUVcfaA}{192.168.99.50}{192.168.99.50:9300}{dil}{ml.machine_memory=2933948416, xpack.installed=true, ml.max_open_jobs=20}, {node-3}{k1BO077zSLm9Iqff1nLqMg}{tEK5LrK9SE2E-s_nO4D3aQ}{192.168.99.10}{192.168.99.10:9300}{ilm}{ml.machine_memory=3897933824, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [192.168.99.10:9300, 192.168.99.40:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0
Log Logstash:
[2020-02-26T23:18:51,096][WARN ][logstash.outputs.elasticsearch][Chamados] DEPRECATION WARNING: Connecting to an OSS distribution of Elasticsearch using the default distribution of Logstash will stop working in Logstash 8.0.0. Please upgrade to the default distribution of Elasticsearch, or use the OSS distribution of Logstash {:url=>"http://192.168.99.50:9200/"}
[2020-02-26T23:18:51,105][WARN ][logstash.outputs.elasticsearch][Chamados] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://192.168.99.40:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://192.168.99.40:9200/, :error_message=>"Elasticsearch Unreachable: [http://192.168.99.40:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-02-26T23:18:51,116][ERROR][logstash.outputs.elasticsearch][Chamados] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://192.168.99.40:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-02-26T23:18:52,131][WARN ][logstash.outputs.elasticsearch][Chamados] Restored connection to ES instance {:url=>"http://192.168.99.40:9200/"}
[2020-02-26T23:19:00,331][INFO ][logstash.inputs.jdbc ][Chamados] (0.003611s) select * from Chamados
Broadcast message from root@elk (Wed 2020-02-26 23:19:05 -03):
The system is going down for power-off at Wed 2020-02-26 23:20:05 -03!
[2020-02-26T23:19:22,221][WARN ][logstash.outputs.elasticsearch][Chamados] DEPRECATION WARNING: Connecting to an OSS distribution of Elasticsearch using the default distribution of Logstash will stop working in Logstash 8.0.0. Please upgrade to the default distribution of Elasticsearch, or use the OSS distribution of Logstash {:url=>"http://192.168.99.40:9200/"}
[2020-02-26T23:20:00,380][INFO ][logstash.inputs.jdbc ][Chamados] (0.003266s) select * from Chamados
Log Kibana:
{"type":"log","@timestamp":"2020-02-27T02:13:41Z","tags":["warning","plugins","licensing"],"pid":2611,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. TypeError: Cannot use 'in' operator to search for 'type' in null"}
{"type":"log","@timestamp":"2020-02-27T02:14:11Z","tags":["warning","plugins","licensing"],"pid":2611,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. TypeError: Cannot use 'in' operator to search for 'type' in null"}
http://192.168.99.40:9200/_cat/health
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
http://192.168.99.10:9200/_cat/indices
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
http://192.168.99.50:9200/_cat/health
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}