Logstash can't finde the certificate

Elastic Stack Logstash
1

Hello, I am trying to set up Logstash but I am failing for a week and now ask for your help. ( Using: /usr/share/logstash/bin/logstash )
I have a certificate chain, which contains the certificate for the webserver and the root CA's. If I set certificate_verification to true it will fail because the certificates in the beats input plugin are self-signed and it, for whatever reason, does not accept the certificate chain..

However, my actual Problem is the Certificate in the Output-Plugin(I guess). By now I understood that the error means the certificate cannot be found. BUT I gave it a wrong path which he immediatly recognized.
Then you may think that logstash simply cannot open the file BUT it fails, even if logstash is the file owner.

I ran out of ideas..

error:

[WARN ] 2021-10-12 12:32:19.957 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://IP:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://IP:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

logstash.yml:

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: $password
xpack.monitoring.elasticsearch.hosts: ["https://IP::9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/server_chained.crt
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true
path.data: /var/lib/logstash
http.host: IP
path.logs: /var/log/logstash

.conf:

input {
        beats{
                host => "IP"
                port => 5044
                ssl => true
                ssl_certificate_authorities => ["/root/testcert/rootCA.crt"]
                ssl_certificate => "/root/testcert/instance.crt"
                ssl_key => "/root/testcert/rootCA.key"
        }
}


output {
        elasticsearch {
                ssl => true
                ssl_certificate_verification => false
                hosts => "IP:9200"
                index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#               cacert => "/etc/logstash/digicert.crt"
                cacert => "/etc/logstash/server_chained.pem"
#               user => "logstash_writer"
#               password => "$password"
        }
}


~                                                                                                                                                            
~                                                                                                                                                            
~
2

Can it be a permissions issue for the file access ?

If the certificate are under another user account and logstash is running under a different account, you have to make sure the certs are accessible.

3 
chmod +r <all files>
chown logstash <all files>

This should do it, right?

I already tried this, if this is what you mean

4

Without the ssl parameters in Output does it work ? What error do you get ?

Also what version are you using ?

It seems your actual issue is the Elasticsearch hosts are not reachable, let alone the certs accessible correctly.

5

Can I just comment them out? Because I get the same error if I just comment these lines out.

Version: 7.14.0

6

I would remove all the SSL references just to see if the Elasticsearch is reachable, then you know the issue is just SSL based if that works.

7

Current .conf:

input {
        beats{
                host => "IP"
                port => 5044
        }
}


output {
        elasticsearch{
                hosts => "https://IP:9200"
                index => "anlagenlog-%{[systeminfo][anlagengruppe]}-%{[systeminfo][anlage]}-%{+YYYY-MM-dd}"
                user => "logstash_writer"
                password => "password"
        }
}

.yml:

path.data: /var/lib/logstash
http.host: IP
path.logs: /var/log/logstash

Still the same error, now I am just confused

8

Same error ?

Elasticsearch Unreachable... removing the host in input ?

input {
        beats{
                host => "IP"
                port => 5044
        }
}
9

Yes, to be precise:

[WARN ] 2021-10-12 13:59:11.693 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@IP:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://logstash_writer:xxxxxx@IP:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
^C[WARN ] 2021-10-12 13:59:15.468 [SIGINT handler] runner - SIGINT received. Shutting down.
^C[FATAL] 2021-10-12 13:59:15.665 [SIGINT handler] runner - SIGINT received. Terminating immediately..

host must be set.

10

WHat happens when you hit the endpoint via a browser with those credentials and is it an Ip address you are using or a IP address? Are those IP addresses and DNS's in the certificate (Common & Alternative names) ?

11

I can use IP and Hostname to access the webbrowser. The credentials are valid aswell, yet I get the same error.
It is strange, isnt it?

12

Have you check the certificates with a SSL tool ? It seems to be the certificates, with the chian and trust.

Browsers are forgiving on the Trust side and not using strict verification.

13

I checked if they are valid and right-structured.
openssl x509 -in <> -text -noout

14

@smam What about

openssl s_client -connect elasticsearch.mydomain.com:443 -prexit -showcerts

or

curl -iv https://elasticsearch.mydomain.com -u username:password

Check your FQDN match what you have in the names matching in "Subject" / "subjectAltName"

* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Dec  2 00:00:00 2020 GMT
*  expire date: Dec  2 23:59:59 2021 GMT
*  subjectAltName: host "elasticsearch.mydomain.com" matched cert's "*.mydomain.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.```
15

I started from scratch and do not run into this error again, so far. The only error I am having now is a non-security one.

Sadly, I cannot tell you or other readers what the problem was but I will thank you for your effort :slight_smile:

16

@smam

Glad it helped in some way to eliminate the problem, now you have the troubleshooting steps when the problem arises again.