smam
October 12, 2021, 10:45am
1
Hello, I am trying to set up Logstash but I am failing for a week and now ask for your help. ( Using: /usr/share/logstash/bin/logstash )
I have a certificate chain, which contains the certificate for the webserver and the root CA's. If I set certificate_verification to true it will fail because the certificates in the beats input plugin are self-signed and it, for whatever reason, does not accept the certificate chain..
However, my actual Problem is the Certificate in the Output-Plugin(I guess). By now I understood that the error means the certificate cannot be found. BUT I gave it a wrong path which he immediatly recognized.
Then you may think that logstash simply cannot open the file BUT it fails, even if logstash is the file owner.
I ran out of ideas..
error:
[WARN ] 2021-10-12 12:32:19.957 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://IP:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://IP:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
logstash.yml:
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: $password
xpack.monitoring.elasticsearch.hosts: ["https://IP::9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/server_chained.crt
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true
path.data: /var/lib/logstash
http.host: IP
path.logs: /var/log/logstash
.conf:
input {
beats{
host => "IP"
port => 5044
ssl => true
ssl_certificate_authorities => ["/root/testcert/rootCA.crt"]
ssl_certificate => "/root/testcert/instance.crt"
ssl_key => "/root/testcert/rootCA.key"
}
}
output {
elasticsearch {
ssl => true
ssl_certificate_verification => false
hosts => "IP:9200"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
# cacert => "/etc/logstash/digicert.crt"
cacert => "/etc/logstash/server_chained.pem"
# user => "logstash_writer"
# password => "$password"
}
}
~
~
~
zx8086
October 12, 2021, 10:57am
2
Can it be a permissions issue for the file access ?
If the certificate are under another user account and logstash is running under a different account, you have to make sure the certs are accessible.
smam
October 12, 2021, 11:14am
3
chmod +r <all files>
chown logstash <all files>
This should do it, right?
I already tried this, if this is what you mean
zx8086
October 12, 2021, 11:34am
4
Without the ssl parameters in Output does it work ? What error do you get ?
Also what version are you using ?
It seems your actual issue is the Elasticsearch hosts are not reachable, let alone the certs accessible correctly.
smam
October 12, 2021, 11:44am
5
Can I just comment them out? Because I get the same error if I just comment these lines out.
Version: 7.14.0
zx8086
October 12, 2021, 11:47am
6
I would remove all the SSL references just to see if the Elasticsearch is reachable, then you know the issue is just SSL based if that works.
smam
October 12, 2021, 12:00pm
7
Current .conf:
input {
beats{
host => "IP"
port => 5044
}
}
output {
elasticsearch{
hosts => "https://IP:9200"
index => "anlagenlog-%{[systeminfo][anlagengruppe]}-%{[systeminfo][anlage]}-%{+YYYY-MM-dd}"
user => "logstash_writer"
password => "password"
}
}
.yml:
path.data: /var/lib/logstash
http.host: IP
path.logs: /var/log/logstash
Still the same error, now I am just confused
zx8086
October 12, 2021, 12:04pm
8
Same error ?
Elasticsearch Unreachable... removing the host in input ?
input {
beats{
host => "IP"
port => 5044
}
}
smam
October 12, 2021, 12:05pm
9
Yes, to be precise:
[WARN ] 2021-10-12 13:59:11.693 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@IP:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://logstash_writer:xxxxxx@IP:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
^C[WARN ] 2021-10-12 13:59:15.468 [SIGINT handler] runner - SIGINT received. Shutting down.
^C[FATAL] 2021-10-12 13:59:15.665 [SIGINT handler] runner - SIGINT received. Terminating immediately..
host must be set.
zx8086
October 12, 2021, 12:09pm
10
WHat happens when you hit the endpoint via a browser with those credentials and is it an Ip address you are using or a IP address? Are those IP addresses and DNS's in the certificate (Common & Alternative names) ?
smam
October 12, 2021, 12:36pm
11
I can use IP and Hostname to access the webbrowser. The credentials are valid aswell, yet I get the same error.
It is strange, isnt it?
zx8086
October 12, 2021, 12:51pm
12
Have you check the certificates with a SSL tool ? It seems to be the certificates, with the chian and trust.
Browsers are forgiving on the Trust side and not using strict verification.
smam
October 12, 2021, 1:18pm
13
I checked if they are valid and right-structured.
openssl x509 -in <> -text -noout
zx8086
October 12, 2021, 3:28pm
14
@smam What about
openssl s_client -connect elasticsearch.mydomain.com:443 -prexit -showcerts
or
curl -iv https://elasticsearch.mydomain.com -u username:password
Check your FQDN match what you have in the names matching in "Subject" / "subjectAltName"
* Server certificate:
* subject: CN=*.mydomain.com
* start date: Dec 2 00:00:00 2020 GMT
* expire date: Dec 2 23:59:59 2021 GMT
* subjectAltName: host "elasticsearch.mydomain.com" matched cert's "*.mydomain.com"
* issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
* SSL certificate verify ok.```
smam
October 14, 2021, 11:45am
15
I started from scratch and do not run into this error again, so far. The only error I am having now is a non-security one.
Sadly, I cannot tell you or other readers what the problem was but I will thank you for your effort
zx8086
October 14, 2021, 2:17pm
16
@smam
Glad it helped in some way to eliminate the problem, now you have the troubleshooting steps when the problem arises again.
1 Like
system
(system)
Closed
November 11, 2021, 2:17pm
17
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.