SSL with Filebeat not working

Elastic Stack Logstash
1

Hi,

I'm trying to secure the traffic between filebeats hosts and the logstash server, my logstash config is;

input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/filebeat/removed.pem"]
ssl_certificate => "/etc/logstash/kibana.internal.removedresource.co.uk.crt"
ssl_key => "/etc/logstash/kibana.internal.removedresource.co.uk.pkcs8.key"
ssk_key_passphrase => "removed"
ssl_verify_mode => "force_peer"
}
}

My filebeat config is;

output.logstash:

The Logstash hosts

ssl.enabled: true
hosts: ["kibana.internal.removedresource.co.uk:5044"]
ssl.certificate_authorities: "/etc/filebeat/removed.pem"
ssl.certificate: "/etc/filebeat/kibana.internal.removedresource.co.uk.crt"
ssl.key: "/etc/filebeat/kibana.internal.removedresource.co.uk.key"

This test passes from the filebeat server;

[root@ewoksagldevap23 filebeat]# curl -v --cacert removed.pem https://kibana.internal.removedresource.co.uk:5044

CApath: none

  • SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

  • Server certificate:

  • subject: CN=kibana.internal.removedresource.co.uk,OU=IT Operations,O=removed PLC,L=Folkestone,ST=Kent,C=GB

  • start date: Jul 17 14:02:35 2020 GMT

  • expire date: Jul 17 14:02:35 2022 GMT

  • common name: kibana.internal.removedresource.co.uk

  • issuer: CN=removedResourceSubCA01,DC=internal,DC=removedresource,DC=co,DC=uk

GET / HTTP/1.1

User-Agent: curl/7.29.0

Host: kibana.internal.removedresource.co.uk:5044

Accept: /

Empty reply from server

Connection #0 to host kibana.internal.removedresource.co.uk left intact

curl: (52) Empty reply from server

However the logging side of logstash loops thought this;

[2020-09-30T15:11:23,257][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x491523f9 run>"}
[2020-09-30T15:11:24,043][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>1.53}
[2020-09-30T15:11:24,376][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2020-09-30T15:11:24,431][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.17}
[2020-09-30T15:11:24,499][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-30T15:11:25,641][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2020-09-30T15:11:26,071][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-09-30T15:11:27,988][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2020-09-30T15:11:28,145][INFO ][logstash.runner ] Logstash shut down.

What am I doing wrong?!? Driving me a little insane :slight_smile:

2

Enable log.level debug in logstash and I expect you will get an additional message that says what the problem is.

3

Thankyou. Good shout.

I can see this in the debug logs;

[2020-09-30T17:30:15,172][DEBUG][logstash.javapipeline ][main] Pipeline terminated by worker error {:pipeline_id=>"main", :exception=>java.lang.IllegalArgumentException: File does not co
ntain valid private key: /etc/logstash/kibana.internal.removedresource.co.uk.pkcs8.key, :backtrace=>["io.netty.handler.ssl.SslContextBuilder.keyManager(io/netty/handler/ssl/SslContextBuilder.j
ava:350)", "io.netty.handler.ssl.SslContextBuilder.forServer(io/netty/handler/ssl/SslContextBuilder.java:107)"

4

I do not know what to say. It needs to be a pkcs8 format key file, and given the file name it is clear you know that. And unless you converted it with the -nocrypt option you need to supply the key password, which you have shown (redacted) in your configuration.

It has come up a couple of times in the last week (probably because the bug about logging the error at debug level was introduced fairly recently). This user got it to work, this one did not (so far).

For testing purposes can you try converting the key with the -nocrypt option and removing

ssk_key_passphrase => "removed"

(I assume that is ssl in your configuration, not ssk, since the plugin would complain if you had a typo in the config.) In fact, you just need to find what it takes to get the plugin started, you do not need anything to connect to it, so you can generate a throw-away cert with nocrypt for the test.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.