Kiaban Web consode access issue - No living connections - ES5.5.2

[Saurabh_Agrawal1 (Saurabh Agrawal)]

Hi All,

I am having 2 EC2 instances running, one is having only ES(Master node) and other Kibana & ES(neither master nor data node).
Status of ES and Kibana application is active/running, but I am not able to open Kibana web console, and getting error message:

Jun 19 16:09:31 amzlinkibt01 kibana[8698]: {"type":"log","@timestamp":"2019-06-19T20:09:31Z","tags":["warning","elasticsearch","admin"],"pid":8698,"message":"Unable to revive connection: https://10.193.72.118:9200/"}
Jun 19 16:09:31 amzlinkibt01 kibana[8698]: {"type":"log","@timestamp":"2019-06-19T20:09:31Z","tags":["warning","elasticsearch","admin"],"pid":8698,"message":"No living connections"}

Need your support to understand what is going wrong and how can I fix it.

Couple of questions I have:

1. In above log message, I want to know from where I am getting admin from, what is it's significance? I have configured default user: elastic and have resetted default password which I am using for any configuration in Kibana and it works.

2. Even when I am not doing request for Kibana web login, I am getting below log messages continiously "Unable to revive connection: https://10.193.72.118:9200/" and "No living connections". Is these messages not for Web login?

3. When I use web to connect to Kibana, I get log messages which is having mention of "statusCode":302.
   Jun 19 16:09:33 amzlinkibt01 kibana[8698]: {"type":"response","@timestamp":"2019-06-19T20:09:33Z","tags":,"pid":8698,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"10.193.72.118:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,hi;q=0.8"},"remoteAddress":"10.255.183.89","userAgent":"10.255.183.89"},"res":{"statusCode":302,"responseTime":4,"contentLength":9},"message":"GET / 302 4ms - 9.0B"}

4. With the default user and resetted password I could install Licence for X-Pack plugin in kibana node. If i use default password, configuration does not happen, so I believe Kibana is able to communicate with ElasticSearch master node. Same License key for X-Pack, I have installed in ES Node.

5. I am using ES 5.5.2, which is getting installed, but I am not able to find error log files and also kibana folder inside /var/log? Does it not happen as part of installation. Though I could refer to /var/log/messages for messages.

6. Snapshot of elasticsearch.yml file in ES Node

cluster.name: elasticsearch-test
node.name: elasticsearch-test-datanode-0
node.master: true
node.data: true
discovery.zen.minimum_master_nodes: 1
gateway.recover_after_nodes: 1
discovery.zen.ping.unicast.hosts: ["10.193.72.219"]

7a. Snapshot of elasticsearch.yml file in Kibana Node
cluster.name: elasticsearch-test
node.name: elasticsearch-test-config
node.master: false
node.data: false
discovery.zen.ping.unicast.hosts: ["10.193.72.219"]
discovery.zen.minimum_master_nodes: 1
gateway.recover_after_nodes: 1

7b. Snapshot of kiaban.yml file in Kibana Node
elasticsearch.requestTimeout: 60000
server.name: "kibana-test"
elasticsearch.url: "https://10.193.72.118:9200"
server.host: 10.193.72.118
elasticsearch.username: elastic
elasticsearch.password: changeme

systemctl status kibana.service -l

● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-06-19 16:07:17 EDT; 2min 17s ago
Main PID: 8698 (node)
CGroup: /system.slice/kibana.service
└─8698 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Jun 19 16:09:31 amzlinkibt01 kibana[8698]: {"type":"log","@timestamp":"2019-06-19T20:09:31Z","tags":["warning","elasticsearch","admin"],"pid":8698,"message":"Unable to revive connection: https://10.193.72.118:9200/"}
Jun 19 16:09:31 amzlinkibt01 kibana[8698]: {"type":"log","@timestamp":"2019-06-19T20:09:31Z","tags":["warning","elasticsearch","admin"],"pid":8698,"message":"No living connections"}

Jun 19 16:09:33 amzlinkibt01 kibana[8698]: {"type":"response","@timestamp":"2019-06-19T20:09:33Z","tags":,"pid":8698,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"10.193.72.118:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,hi;q=0.8"},"remoteAddress":"10.255.183.89","userAgent":"10.255.183.89"},"res":{"statusCode":302,"responseTime":4,"contentLength":9},"message":"GET / 302 4ms - 9.0B"}

[Marius_Dragomir]

hello,

Is the ES instance accessible via the console from the Kibana EC2 instance?