SSL between Filebeat and Logstash using .p12

Following the documenation (https://www.elastic.co/guide/en/beats/filebeat/6.1/configuring-ssl-logstash.html) it shows how to configure Filebeat and Logstash. But when reading the documenation for creating certificates using certutil , it shows how to make a .p12 (which includes the certificate and key if I understand it correctly). So how do you configure Filebeat and Logstash to use the .p12 certificate or is that not possible?

output.logstash:
  hosts: ["logs.mycompany.com:5044"]
  ssl.certificate_authorities: ["/etc/ca.crt"]
  ssl.certificate: "/etc/client.crt"
  ssl.key: "/etc/client.key"
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/ca.crt"]
    ssl_certificate => "/etc/server.crt"
    ssl_key => "/etc/server.key"
    ssl_verify_mode => "force_peer"
  }
}

Hi there, certutil is only intended for use with Elasticsearch. For Logstash and Filebeat you'll need to extract the PEM-style certificate and key from your PKCS12.

Thanks,
CJ

certutil has a -pem option to produce certificates in PEM format that you can use in beats.

Thank you for the answers, I was able to get the key and certificates but now logstash is giving me an error.

[2018-04-11T10:11:40,270][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@localhost:9200/]}}
[2018-04-11T10:11:40,271][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-04-11T10:11:40,279][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@localhost:9200/"}
[2018-04-11T10:11:40,290][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[2018-04-11T10:11:40,291][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-04-11T10:11:40,291][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-04-11T10:11:40,294][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-04-11T10:11:40,304][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-04-11T10:11:40,600][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x13d13cf run>"}
[2018-04-11T10:11:40,605][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-04-11T10:11:40,611][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Beats port=>5044, ssl=>true, ssl_certificate_authorities=>[\"C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/cert/ca.crt\"], ssl_certificate=>\"C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/cert/instance.crt\", ssl_key=>\"C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/cert/instance.key\", ssl_verify_mode=>\"force_peer\", id=>\"c7ccdd57fa15ab8f83753208a1455a3551ac4dc202e4fdc76397396bd58afdb8\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_518dfb31-8772-45dd-8137-8134b19b10a2\", enable_metric=>true, charset=>\"UTF-8\">, host=>\"0.0.0.0\", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>[\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\"], client_inactivity_timeout=>60, executor_threads=>16>", :error=>"Cipher `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384` is not available", :thread=>"#<Thread:0x13d13cf run>"}
[2018-04-11T10:11:41,298][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Cipher `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384` is not available>, :backtrace=>["C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.6-java/lib/logstash/inputs/beats.rb:170:in `create_server'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.6-java/lib/logstash/inputs/beats.rb:158:in `register'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:343:in `register_plugin'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:354:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:354:in `register_plugins'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:510:in `start_inputs'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:401:in `start_workers'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:288:in `run'", "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x13d13cf run>"}
[2018-04-11T10:11:41,301][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
input {
    beats {
        port => "5044"
		ssl => true
		ssl_certificate_authorities => ["C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/cert/ca.crt"]
		ssl_certificate => "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/cert/instance.crt"
		ssl_key => "C:/Users/jordyd/Documents/selfhostedsearch/ELK/logstash-6.1.3/cert/instance.key"
		ssl_verify_mode => "force_peer"
    }
}

Try here: Logstash TLS with x-pack error registering plugin (Cipher is not available) 6.x

Okay, the above link helped but it seems my certificates I generated with the x-pack are bad. I generated the file using the following command "certutil ca --pem" and this gave me a zip with the CA. Then I created one "certutil cert (nameofzip) -pem". I copied the cert,key and CA to a folder for both Logstash and Filebeat.

Filebeat

2018/04/11 11:09:48.453351 output.go:74: ERR Failed to connect: x509: cannot validate certificate for 10.203.207.76 because it doesn't contain any IP SANs

Logstash

[2018-04-11T13:06:48,425][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.203.204.70:35542] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000412:SSL

The default options for certutil are designed to work with TLS for inter-cluster communication. If you want to generate certificates for another purpose, then you often need to apply different options.

By default beats will require that Logstash present a certificate with the correct hostname (SAN) for the address to which beats connects.

When you generate the Logstash certificate with certutil you should pass -ip 10.203.207.76

I created the certificates, but this time with the ip adress and now logstash is giving me this error.

[2018-04-12T12:51:21,323][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.203.207.127:50626] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Available ciphers:[ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-CHACHA20-POLY1305, ECDHE-RSA-CHACHA20-POLY1305, ECDHE-PSK-CHACHA20-POLY1305, ECDHE-ECDSA-AES128-SHA, ECDHE-ECDSA-AES128-SHA256, ECDHE-RSA-AES128-SHA, ECDHE-RSA-AES128-SHA256, ECDHE-PSK-AES128-CBC-SHA, ECDHE-ECDSA-AES256-SHA, ECDHE-ECDSA-AES256-SHA384, ECDHE-RSA-AES256-SHA, ECDHE-RSA-AES256-SHA384, ECDHE-PSK-AES256-CBC-SHA, AES128-GCM-SHA256, AES256-GCM-SHA384, AES128-SHA, AES128-SHA256, PSK-AES128-CBC-SHA, AES256-SHA, AES256-SHA256, PSK-AES256-CBC-SHA, DES-CBC3-SHA]
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Ciphers:  [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256]
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Certificate Authorities: [/etc/logstash/cert/ca.crt]
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Load certificates collection
[2018-04-12T12:51:22,327][DEBUG][org.logstash.netty.SslSimpleBuilder] Loading certificates from file /etc/logstash/cert/ca.crt
[2018-04-12T12:51:22,341][DEBUG][org.logstash.netty.SslSimpleBuilder] TLS: [TLSv1, TLSv1.1, TLSv1.2]
[2018-04-12T12:51:22,357][DEBUG][logstash.inputs.beats    ] Cannot retrieve remote IP address for beats input - remoteAddress is nil
[2018-04-12T12:51:22,359][DEBUG][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1120) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1080) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1146) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1189) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1248) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1171) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1196) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	... 16 more
[2018-04-12T12:51:22,360][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.