No living connections error in kibana after enabling TLS/SSL

Elastic Stack Kibana
1

Hi Folks,

I think I might be missing something obvious but I am attempting to enable basic security on our Testing ELK Stack. enabling SSL on elasticsearch was successful. The certificate I am using is restricted to accept the following:
localhost
kibana.local
logstash.local
elastic.local
I have used certutil cert --ca {ca file path} --pem to create a certificate in pem format for Kibana and Logstash. The ca certificate does have a password.

I have set the hostname on kibana.yml to kibana.local with elasticsearch https://elastic.local:9200

I can connect using elastichead to this url and I can also connect using powershell
I have set the certificate settings in kibana.yml to
certificate (pem format) {filepath/instance.crt}
key {filepath/instance.key}
I have also created and added a kibana keystore for the credentials to connect to elasticsearch.

If I set the verification to None it connects ok but when using Certificate I get the error. This leads me to think it is a certificate issue but I don't know how to confirm suspicions.

Any advice or guidance would be much appreciated.

2

Which version of the Elastic stack are you using?

Is Logstash working?

Did you set elasticsearch.ssl.certificate and elasticsearch.ssl.key?

If you're still having trouble it might be helpful to post your kibana.yml and elasticsearch.yml files.

3

Hi Mattkime,

Thank you for your response.

I have not tried logstash just yet. I was getting kibana working so I can copy the config over.

I have set elasticsearch.ssl.certificate and .key without success. I do not have openssl available due to our security policy.

My Elasticsearch config is

bootstrap.memory_lock: false
cluster.name: elasticsearch
http.port: 9200
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: elastic.local
network.host: [elastic.local,local]
path.data: elasticsearch\data
path.logs: elasticsearch\logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/node-1.p12
xpack.security.transport.ssl.truststore.path: certs/node-1.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/node-1.p12
xpack.security.http.ssl.truststore.path: certs/node-1.p12
discovery.type: single-node

My Kibana config is

server.host: "localhost"
elasticsearch.hosts: ["https://127.0.0.1:9200"]
elasticsearch.preserveHost: true
server.ssl.enabled: true
server.ssl.certificate: 'certs\ca.crt'
server.ssl.key: 'certs\ca.key'
elasticsearch.ssl.certificateAuthorities: 'certs\cacert.pem'
elasticsearch.ssl.verificationMode: certificate
elasticsearch.logQueries: true
logging.dest: logs\kibana-logs.log
logging.verbose: true
logging.json: true
xpack.security.enabled: true
xpack.security.encryptionKey: "32 character string"

If you set the verificationmode to none Kibana connects ok to Elasticsearch but when setting to certificate I see:

{"type":"log","@timestamp":"2019-08-27T23:56:24Z","tags":["warning","elasticsearch","admin"],"pid":11572,"message":"Unable to revive connection: https://127.0.0.1:9200/"}
{"type":"log","@timestamp":"2019-08-27T23:56:24Z","tags":["warning","elasticsearch","admin"],"pid":11572,"message":"No living connections"}

It could something really simple but I cannot see it

4

I have now managed to convert the certificate to .pem which has allowed kibana to connect but now I am seeing the following error:

Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

What can cause this?

5

Ok so I am a bit further on now and I have encountered another error which seems to imply it is a java error. My Knowledge of Java is non existent and the error is:
[2019-08-29T13:38:43,822][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_internal:xxxxxx@elastic.local:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_internal:xxxxxx@elastic.local:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

The self signed certificate is installed in the trusted on the local machine. If you know what is causing the issue and how to resolve let me know.

my logstash config is

output {
elasticsearch {
hosts => ["https://elastic.local:9200"]
index => "logstash-%{+YYYY.MM.dd}"
ssl => true
keystore => '\config\logstash.keystore'
keystore_password => "keystore.pass"
cacert => '\config\certs\instance.crt'
ssl_certificate_verification => true
user => "${ES_USER}"
password => "${ES_PWD}"
}
stdout { codec => rubydebug }
}

6

@cowensel

Thanks for the updates and I'm glad to see you're making progress. I will try to replicate the error you're seeing but it might take me a couple of days.

7

Hi Mattkime,

I have now managed to resolve the problem, To do this I browsed to the url in the error message:
https://logstash_internal:xxxxxx@elastic.local:9200/ and exported the certificate and pointed the cacert to the exported certificate which resolved the issue.

The documentation I had that step missing.

Thanks for your assistance in getting this resolved

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.