# Kibana can not connect to Elasticsearch cluster **URL:** \ **Category:** Kibana\ **Created:** [June 16, 2020, 4:00pm UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335 "2020-06-16T16:00:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![skeleton](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@skeleton](https://discuss.elastic.co/u/skeleton)\ **Post date:** [June 16, 2020, 4:00pm UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/1 "2020-06-16T16:00:22Z")
First and foremost: Apologies, if this post is poorly formatted, i am unfamiliar with this editor. I will try to fix it once i see it, because i can not find the preview function. Background: I am trying to get a Wazuh implementation running. Feel free to point me to their forum if this issue is a Wazuh-related problem, but it seems to me like this might be the better place. I am trying to get my Kibana Server to talk to my Elasticsearch Server. I had configured a setup with Kibana and Elasticsearch on one server before which worked fine. Now my **Kibana server can not connect to my Elasticsearch server via port 9200** and my **cluster of three Elasticsearch servers can not decide who the master server should be.** I will start with the Kibana problem so i do not go over the character limit: Here is the log output from my **Kibana Server** : ```auto {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","config","deprecation"],"pid":1490,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana\" user instead."} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","config","deprecation"],"pid":1490,"message":"Setting [xpack.monitoring.elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana\" user instead."} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins-system"],"pid":1490,"message":"Setting up [37] plugins: [taskManager,siem,licensing,infra,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,r eporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","taskManager"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","siem"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","licensing"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","infra"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","encryptedSavedObjects"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":1490,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","code"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","usageCollection"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","metrics"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","canvas"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","timelion"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","features"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","security"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","security","config"],"pid":1490,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","security","config"],"pid":1490,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","apm_oss"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","translations"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","data"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","share"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","home"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","spaces"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","cloud"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","apm"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","graph"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","bfetch"],"pid":1490,"message":"Setting up plugin"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","savedobjects-service"],"pid":1490,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","elasticsearch","data"],"pid":1490,"message":"Request error, retrying\nGET http://192.168.1.91:9200/_xpack => socket hang up"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","elasticsearch","data"],"pid":1490,"message":"Request error, retrying\nHEAD http://192.168.1.91:9200/.apm-agent-configuration => socket hang up"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","elasticsearch","admin"],"pid":1490,"message":"Request error, retrying\nGET http://192.168.1.91:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => socket hang up"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","licensing"],"pid":1490,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","savedobjects-service"],"pid":1490,"message":"Unable to retrieve version information from Elasticsearch nodes."} {"type":"log","@timestamp":"2020-06-16T14:21:52Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:52Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:21:54Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:54Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:21:57Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:57Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:21:59Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} {"type":"log","@timestamp":"2020-06-16T14:21:59Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"} {"type":"log","@timestamp":"2020-06-16T14:22:02Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"} ``` Here is the output of `netstat -tulpen` on the elasticsearch master node ```auto Proto Recv-Q Send-Q Local Address Foreign Address State Benutzer Inode PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 13822 407/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 14202 563/master tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 109 13734 336/nrpe tcp6 0 0 192.168.1.91:9200 :::* LISTEN 111 37614 2610/java tcp6 0 0 192.168.1.91:9300 :::* LISTEN 111 37480 2610/java tcp6 0 0 ::1:25 :::* LISTEN 0 14203 563/master udp 0 0 127.0.0.1:323 0.0.0.0:* 0 13539 348/chronyd udp 0 0 0.0.0.0:41297 0.0.0.0:* 0 13009 310/rsyslogd ``` Here is what i get, when i try to open a telnet connection from the kibana server to the elasticsearch master node with `telnet 192.168.1.91 9200`. It looks to me like the port is open. ```auto Trying 192.168.1.91... Connected to 192.168.1.91. Escape character is '^]'. ``` Any help would be greatly appreciated! If you need further input from me, let me know. Thanks! ---
**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\ **Post date:** [June 16, 2020, 4:55pm UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/2 "2020-06-16T16:55:02Z")
Hello @skeleton - can you post your elasticsearch.yml and kibana.yml ? ---
**Author:** ![skeleton](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@skeleton](https://discuss.elastic.co/u/skeleton)\ **Post date:** [June 17, 2020, 11:53am UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/3 "2020-06-17T11:53:47Z")
Of course, thank you for taking a look. I have tried commenting out the two slave nodes under `cluster.initital_master_nodes:` I have also tried setting all options for XPACK Security from true to false in all Elasticsearch nodes, but that leaves me with a different error message from kibana complaining about a timeout. Let me know if i should post that. **elasticsearch.yml of the server intended to be the master node** ```auto # Ansible managed cluster.name: wazuh node.name: elastic-os-01 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.1.91 discovery.zen.ping.unicast.hosts: ["192.168.1.91", "192.168.1.92", "192.168.1.93"] node.master: true node.data: true cluster.initial_master_nodes: - 192.168.1.91 # - 192.168.1.92 # - 192.168.1.93 discovery.seed_hosts: - 192.168.1.91 - 192.168.1.92 - 192.168.1.93 # XPACK Security xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elastic-os-01.key xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elastic-os-01.crt xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"] xpack.security.http.ssl.enabled: true xpack.security.http.ssl.verification_mode: certificate xpack.security.http.ssl.key: /etc/elasticsearch/certs/elastic-os-01.key xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elastic-os-01.crt xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"] ``` **elasticsearch.yml of one the other two nodes** ```auto # Ansible managed cluster.name: wazuh node.name: elastic-os-03 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.1.93 discovery.zen.ping.unicast.hosts: ["192.168.1.91", "192.168.1.92","192.168.1.93"] node.master: false node.data: true cluster.initial_master_nodes: - 192.168.1.91 discovery.seed_hosts: - 192.168.1.91 - 192.168.1.92 - 192.168.1.93 # XPACK Security xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elastic-os-03.key xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elastic-os-03.crt xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"] xpack.security.http.ssl.enabled: true xpack.security.http.ssl.verification_mode: certificate xpack.security.http.ssl.key: /etc/elasticsearch/certs/elastic-os-03.key xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elastic-os-03.crt xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"] ``` ---
**Author:** ![skeleton](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@skeleton](https://discuss.elastic.co/u/skeleton)\ **Post date:** [June 17, 2020, 11:54am UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/4 "2020-06-17T11:54:42Z")
**kibana.yml** ```auto # Ansible managed # Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: 0.0.0.0 # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests # to Kibana. This setting cannot end in a slash. #server.basePath: "" # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URL of the Elasticsearch instance to use for all your queries. elasticsearch.hosts: "http://192.168.1.91:9200" # When this setting's value is true Kibana uses the hostname specified in the server.host # setting. When the value of this setting is false, Kibana uses the hostname of the host # that connects to this Kibana instance. #elasticsearch.preserveHost: true # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. kibana.index: ".kibana" # The default application to load. #kibana.defaultAppId: "discover" # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. #elasticsearch.username: "elastic" #elasticsearch.password: "elastic_pass" # Paths to the PEM-format SSL certificate and SSL key files, respectively. These # files enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.cert: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files validate that your Elasticsearch backend uses the same key files. #elasticsearch.ssl.cert: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.ca: /path/to/your/CA.pem # To disregard the validity of SSL certificates, change this setting's value to false. #elasticsearch.ssl.verify: true # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [authorization] # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 0 # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying. #elasticsearch.startupTimeout: 5000 # Specifies the path where Kibana creates the process ID file. #pid.file: /var/run/kibana.pid # Enables you specify a file where Kibana stores log output. logging.dest: /usr/share/kibana/kibana.log # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Xpack Security ``` ---
**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\ **Post date:** [June 18, 2020, 3:33pm UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/5 "2020-06-18T15:33:48Z")
Hello @skeleton - I think we should first focus on getting you ES cluster running. What is the current error you're seeing? It might be good to start with a single ES instance and then add the two additional instances. ---
**Author:** ![skeleton](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@skeleton](https://discuss.elastic.co/u/skeleton)\ **Post date:** [July 3, 2020, 1:46pm UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/6 "2020-07-03T13:46:00Z")
Hello Matt, sorry for disappearing for so long. Maybe it will make you at least little happy that we managed to fix the issue ourselves: The problem with the elasticsearch servers being unable to elect a Master server stopped when we edited the **elasticsearch.yml** of the second node and set ```auto node.master: false ``` to ```auto node.master: true ``` Kibana then happily connected too. Thank you for providing support to novices like me. I hope you have a good weekend! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 31, 2020, 1:46pm UTC](https://discuss.elastic.co/t/kibana-can-not-connect-to-elasticsearch-cluster/237335/7 "2020-07-31T13:46:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.