Logstash SSL verification

hanna · January 6, 2023, 11:53am

Hello,

how can I enable SSL certificate verification in my logstash pipeline output to elasticsearch? I don't find any documentation on which certificates to use here.
The cluster version is 8.5.3.

This is my logstash pipeline:

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

output {
  elasticsearch {
    index => "logstash-init-%{+YYYY.MM.dd}"
    ssl => true
    cacert => "/etc/logstash/certs/http_ca.crt"
    ssl_certificate_verification => true
    hosts => ['ELASTIC-01', 'ELASTIC-02', 'ELASTIC-03']
    user => "logstash_writer"
    password => "password"
  }
}

I am getting the following error message:

Jan 6 12:20:30 LOGSTASH-01 logstash[18785]: [2023-01-06T12:20:30,964][WARN ][logstash.outputs.elasticsearch][init_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@ELASTIC-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://ELASTIC-01:9200/][Manticore::ClientProtocolException] PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors"}

Everything works fine as long as the ssl_certificate_verification is set to false - cluster is healthy, kibana enrolls and pipeline works as intended.
I have configured elasticsearch to use transport certificates generated within my companies PKI and I am also using the auto generated http.p12 keystore and http_ca.crt. As far as I understand logstash uses the elasticsearch http port and not the transport one so the certificate I need for logstash to trust elasticsearch would be the ca certificate from the http configuration "http_ca.crt". However this does not work and I am getting the error message from above.
What did I get wrong? What certificate file in what format does logstash need to establish trust and where do I get it from??

The ssl section from my elasticsearch.yml:

...
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.key: certs/ES-1.pem
xpack.security.transport.ssl.certificate: certs/ES-1.pem
xpack.security.transport.filter.allow: [ "localhost", 172.16.10.1/24 ]
xpack.security.transport.filter.deny: _all
...

Here is an illustration of my certification configuration:

I appreciate any help, I've been tinkering with this for a few days now and I'm very confused at this point

dadiasish · January 6, 2023, 12:53pm

Give the above cert in the logstash output and try.

That should work.

Ayush_Mathur · January 6, 2023, 1:47pm

@hanna may be try following the blog post from Elastic to configure TLS between stack components: Configuring SSL, TLS, and HTTPS to secure Elasticsearch, Kibana, Beats, and Logstash | Elastic Blog

hanna · January 6, 2023, 2:23pm

Thank you for your answer @dadiasish.
I had already tried using this certificate but it throws this error:

Jan  6 14:41:28 LOGSTASH-01 logstash[21022]: [2023-01-06T14:41:28,173][WARN ][logstash.outputs.elasticsearch][init_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@ELASTIC-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://ELASTIC-01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
Jan  6 14:41:28 LOGSTASH-01 logstash[21022]: [2023-01-06T14:41:28,198][INFO ][logstash.outputs.elasticsearch][init_pipeline] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}

The elasticsearch logs are giving me this warning, when I'm using the certificate:

[2023-01-06T15:20:35,414][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [ELASTIC-01] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/172.16.10.55:9200, remoteAddress=/172.16.3.25:58392}

dadiasish · January 6, 2023, 2:36pm

Do you have all the permissions to the respective path where your certificate is present?

Give the necessary permissions to the folder and also make the cert file executable file and try running.

And let me know if it worked.

hanna · January 9, 2023, 9:07am

Thank you @Ayush_Mathur.
I can't reproduce all the steps from the blog, because my setup is a bit different. But I tried adapting the instructions and step 4-3 actually fixed my logstash issue!

my new elasticsearch.yml:

...
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
#  keystore.path: certs/http.p12
xpack.security.http.ssl.key: certs/elastic.key
xpack.security.http.ssl.certificate: certs/elastic.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.key: certs/elastic.key
xpack.security.transport.ssl.certificate: certs/elastic.crt
...

However this kills the connection between kibana and elasticsearch as I need the http.p12 keystore for the kibana enrollment. (I am using enrollment tokens to automatically enroll kibana)

But thats another issue and I guess if I just manually connect Kibana everything should finally work!

hanna · January 9, 2023, 9:15am

Thanks for your advice, the problem was in the elasticsearch certificate configuration. It is now solved

Ayush_Mathur · January 9, 2023, 11:15am

Good to hear that

system · February 6, 2023, 11:16am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.