"PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed"

Elastic Stack Logstash
1

Here's the full error:

Oct 07 11:57:38 elk.example.com logstash[3697608]: [2024-10-07T11:57:38,571][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_internal:xxxxxx@elk.example.com:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elk.example.com:9200/][Manticore::ClientProtocolException] PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed"}

It was working fine till about 09:45 today morning after which it started showing up with this. I use the same ca.crt for both Elasticsearch and Logstash and have chowned and setfacled the respective directories (I do not think this is a problem with permissions). ca.crt is valid till 2035.

Can someone help me? I have seen many similar errors but this specifically never turned up in my search.

The only things I have changed from the default logstash.yml are:

  •    path.data: /usr/share/logstash/data
  • pipeline.id: main

Thanks

Edit: Relevant part of my pipeline config

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["https://elk.example.com:9200"]
    index => "%{log_index}"
    action => "create"
    ssl_certificate_authorities => "/etc/logstash/ssl/ca.crt"
    ssl => true
    user => "logstash_internal"
    password => <some password>
  }
}
2

What timezone are you in?

The most common reason for that error seems to be that one of the certificates in the validation chain has expired.

3

Hello, I'm in EST in the US.

You're probably right. A few other things broke in our infrastructure for similar reasons and looks like the cert is the problem. I will be investigating this today, will get back once I (attempt to) fix this.

Thanks

4

Unfortunately, that doesn't seem to have fixed the problem. I replaced the cert with a new cert (the previous one had already expired). Both of these certs are derived from an intermediate certificate.

What am I missing? This is my Elasticsearch config:

xpack.security.http.ssl:
  enabled: true
  key: ssl/elk.example.com.key
  certificate: ssl/elk.example.com.crt
  certificate_authorities: ssl/ca.crt

xpack.security.transport.ssl:
  enabled: true
  verification_mode: none
  key: ssl/elk.example.com.key
  certificate: ssl/elk.example.com.crt
  certificate_authorities: ssl/ca.crt

(ssl is a subdirectory in /etc/elasticsearch and elk.example.com.crt has been replaced by a new cert which is valid. I did not rotate the private key.

Edit: complete error log from Logstash:

Oct 08 09:20:21 elk.example.com logstash[4120966]: [2024-10-08T09:20:21,670][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Invalid CertificateVerify signature", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: Invalid CertificateVerify signature>}

Oct 08 09:20:21 elk.example.com logstash[4120966]: [2024-10-08T09:20:21,671][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_internal:xxxxxx@elk.example.com:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elk.example.com:9200/][Manticore::ClientProtocolException] Invalid CertificateVerify signature"}
5

Hello,

It seems that there was a problem in the certificate that was generated. I checked the certificate with openssl and was seeing Validation Errors. We produced a new certificate for Elasticsearch and now everything is working fine. Hope this helps anyone who comes across this thread in the future.