How to resolve 'Unable to revive connection'?

Muhammad_Ariful_Isla · October 10, 2019, 7:42pm

I have a Elasticsearch instance in dev mode. My Elasticsearch config is following,

cluster.name: "logparser"
node.master: true
node.data: true
node.ingest: true
node.ml: false
path.data: /storage/elasticsearch
path.logs: /var/log/elasticsearch
network.host: _local_
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elasticsearch.p12
xpack.security.http.ssl.truststore.path: certs/elasticsearch.p12

Following is my Kibana configuration,

server.port: 5601
server.host: 172.16.10.10
server.name: "logparser.akash.pub"
elasticsearch.hosts: ["https://127.0.0.1:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "blabla"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elastic-ca.pem" ]
xpack.security.enabled: true

For your information, I have created .pem file from the kibana.p12 file which created from elastic-stack-ca.p12

My Problem is ,

Oct 11 01:24:19 logparser kibana: {"type":"log","@timestamp":"2019-10-10T19:24:19Z","tags":["warning","elasticsearch","admin"],"pid":3862,"message":"Unable to revive connection: https://127.0.0.1:9200/"}
Oct 11 01:28:18 logparser kibana: {"type":"log","@timestamp":"2019-10-10T19:28:18Z","tags":["warning","elasticsearch","admin"],"pid":3862,"message":"No living connections"}

I can reach Elasticsearch with the Kibana's credential,

curl -kA "Mozilla" -L "https://kibana:blabla@localhost:9200"                                                                                                                                
{
"name" : "logparser.akash.pub",
"cluster_name" : "logparser",
"cluster_uuid" : "eR48rtysQHWTa89ZxXBrNQ",
"version" : {
"number" : "7.4.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "22e1767283e61a198cb4db791ea66e3f11ab9910",
"build_date" : "2019-09-27T08:36:48.569419Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Current network status,

[root@logparser]# netstat -tulpn 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 172.16.10.10:5601       0.0.0.0:*               LISTEN      3862/node           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1135/sshd           
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      3370/java           
tcp6       0      0 ::1:9200                :::*                    LISTEN      3370/java           
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      3370/java           
tcp6       0      0 ::1:9300                :::*                    LISTEN      3370/java           
tcp6       0      0 :::22                   :::*                    LISTEN      1135/sshd

How to resolve the problem?

tylersmalley · October 10, 2019, 8:16pm

Can you provide all the Kibana logs up to "Unable to revive connections" I wonder if we're missing a log entry related to the issue here.

My guess it's related to the certificate authority. If so, we can test that theory by setting elasticsearch.ssl.verificationMode: none in the kibana.yml file.

Muhammad_Ariful_Isla · October 10, 2019, 8:51pm

As I can't dump that much log in here, I have just filtered out the plugin logs. Here is log from the beginning of kibana service and upto Unable to revive

Oct 10 14:06:38 logparser systemd: Started Kibana.
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["info","siem"],"pid":10376,"message":"Plugin initializing"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["info","siem"],"pid":10376,"message":"Plugin done initializing"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["warning","encrypted_saved_objects"],"pid":10376,"message":"Generating a random key for xpack.encrypted_saved_objects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encrypted_saved_objects.encryptionKey in kibana.yml"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["reporting","browser-driver","warning"],"pid":10376,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["error","elasticsearch","admin"],"pid":10376,"message":"Request error, retrying\nHEAD http://localhost:9200/.apm-agent-configuration => socket hang up"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["error","elasticsearch","admin"],"pid":10376,"message":"Request error, retrying\nGET http://localhost:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => socket hang up"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["warning","elasticsearch","admin"],"pid":10376,"message":"Unable to revive connection: http://localhost:9200/"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["warning","elasticsearch","admin"],"pid":10376,"message":"No living connections"}
Oct 10 14:06:51 logparser kibana: Could not create APM Agent configuration: No Living connections
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["warning","elasticsearch","admin"],"pid":10376,"message":"Unable to revive connection: http://localhost:9200/"}
Oct 10 14:06:51 logparser kibana: {"type":"log","@timestamp":"2019-10-10T08:06:51Z","tags":["warning","elasticsearch","admin"],"pid":10376,"message":"No living connections"}

Muhammad_Ariful_Isla · October 10, 2019, 9:21pm

And enabling elasticsearch.ssl.verificationMode: none results positive i think,
I have deleted the .kibana* indices from elasticsearch. After restarting the service i saw those indices again and no Unable to revive. I will try to regenerate the certificates again.

[root@logparser ~]# curl -XGET http://localhost:5601/status -I
HTTP/1.1 302 Found
location: /login?next=%2Fstatus
kbn-name: kibana
kbn-xpack-sig: b8ff2177f2128fa9b24bd9ded93eecc2
cache-control: no-cache
content-length: 0
Date: Thu, 10 Oct 2019 21:15:29 GMT
Connection: keep-alive

And the last line of kibana log,

Oct 11 03:11:03 logparser kibana: {"type":"log","@timestamp":"2019-10-10T21:11:03Z","tags":["info","http","server","Kibana"],"pid":3382,"message":"http server running at http://0.0.0.0:5601"}
Oct 11 03:11:04 logparser kibana: {"type":"log","@timestamp":"2019-10-10T21:11:04Z","tags":["status","plugin:spaces@7.4.0","info"],"pid":3382,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

system · November 7, 2019, 9:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.