Filebeat 6.1.1 not connecting to logstash over ssl


(Greg Sullivan) #1

I am having an issue with my ssl connection to logstash, Filebeat says everything is ok,

  C:\filebeat>filebeat test output
logstash: logstash:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xx.xx.xx.xx
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK

ran >filebeat -c filebeat.yml -e -d "*"
didn't show any issues.

But on Logstash I see the connection was closed,

**[2018-01-19T18:21:47,080][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.**
**0:5044, remote: undefined] Exception: An existing connection was forcibly closed**
** by the remote host,**

filebeat.yml

filebeat.prospectors:

- type: log

   enabled: true
 
  paths:
    #- /var/log/*.log
    - c:\fileuploads\*
name: Filebeat1
output.logstash:
  # The Logstash hosts
  hosts: ["logstash:5044"]
  protocol: "https"
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["c:\\certs\\ca\\ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "c:\\certs\\filebeat1\\filebeat1.crt"

  # Client Certificate Key
  ssl.key: "c:\\certs\\filebeat1\\filebeat1.key"

logstash beats config,

input { 
	beats {
	port => 5044
	codec => "json"
	ssl => true
	ssl_certificate_authorities =>	["D:\logstash-6.1.1\config\ca\ca.crt"]
	ssl_certificate => "D:\logstash-6.1.1\config\logstash\logstash.crt"
	ssl_key => "D:\logstash-6.1.1\config\logstash\logstash.key"	
	} 
 }

any ideas?


(Pier-Hugues Pellerin) #2

@GSCully Can you start logstash in with debug turn on? bin/logstash --log.level debug This should give us a bit more idea what is happening.


(Greg Sullivan) #3

thankjs, it appears a cert is bad?

[2018-01-19T21:26:42,822][DEBUG][org.logstash.netty.SslSimpleBuilder] Certificate Authorities: [D:\logstash-6.1.1\config\ca\ca.crt]
[2018-01-19T21:26:42,822][DEBUG][org.logstash.netty.SslSimpleBuilder] Load certificates collection
[2018-01-19T21:26:42,822][DEBUG][org.logstash.netty.SslSimpleBuilder] Loading certificates from file D:\logstash-6.1.1\config\ca\ca.crt
[2018-01-19T21:26:42,838][DEBUG][org.logstash.netty.SslSimpleBuilder] Available ciphers:...
[2018-01-19T21:26:42,854][DEBUG][org.logstash.netty.SslSimpleBuilder] Certificate Authorities: [D:\logstash-6.1.1\config\ca\ca.crt]
[2018-01-19T21:26:42,854][DEBUG][org.logstash.netty.SslSimpleBuilder] Load certificates collection
[2018-01-19T21:26:42,854][DEBUG][org.logstash.netty.SslSimpleBuilder] Loading certificates from file D:\logstash-6.1.1\config\ca\ca.crt
[2018-01-19T21:26:42,869][DEBUG][org.logstash.netty.SslSimpleBuilder] TLS: [TLSv1, TLSv1.1, TLSv1.2]
[2018-01-19T21:26:42,869][DEBUG][org.logstash.netty.SslSimpleBuilder] TLS: [TLSv1, TLSv1.1, TLSv1.2]
[2018-01-19T21:26:42,885][DEBUG][io.netty.util.internal.JavassistTypeParameterMatcherGenerator] Generated: io.netty.util.internal.matchers.org.logstash.beats.AckMatcher
[2018-01-19T21:26:42,885][DEBUG][io.netty.util.internal.JavassistTypeParameterMatcherGenerator] Generated: io.netty.util.internal.matchers.org.logstash.beats.BatchMatcher
[2018-01-19T21:26:42,947][DEBUG][io.netty.handler.ssl.OpenSslEngine] SSL_read failed: OpenSSL error: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2018-01-19T21:26:42,979][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Exception: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2018-01-19T21:26:42,979][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Exception: An existing connection was forcibly closed by the remote host

Would that be the ca cert? or the logstash cert or beats? I suspect beats.


(Pier-Hugues Pellerin) #4

I suspect beats too, let's check if you can do server auth only without doing client auth.

Remove the following lines:

 # Certificate for SSL client authentication
  ssl.certificate: "c:\\certs\\filebeat1\\filebeat1.crt"

  # Client Certificate Key
  ssl.key: "c:\\certs\\filebeat1\\filebeat1.key"

(Greg Sullivan) #5

I don't seethe same errors but I also don't see a connection,
[io.netty.util.internal.JavassistTypeParameterMatcherGenerator] Generated: io.netty.util.internal.matchers.org.logstash.beats.AckMatcher
[2018-01-19T22:15:00,879][DEBUG][io.netty.util.internal.JavassistTypeParameterMatcherGenerator] Generated: io.netty.util.internal.matchers.org.logstash.beats.BatchMatcher
[2018-01-19T22:15:02,847][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x2a9aab53 sleep>"}
[2018-01-19T22:15:07,425][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Exception: not an SSL/TLS record: 3f0d0a650d0a
[2018-01-19T22:15:07,832][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["D:/logstash-6.1.1/CONTRIBUTORS", "D:/logstash-6.1.1/Gemfile", "D:/logstash-6.1.1/Gemfile.lock", "D:/logstash-6.1.1/LICENSE", "D:/logstash-6.1.1/NOTICE.TXT", "D:/logstash-6.1.1/bin", "D:/logstash-6.1.1/config", "D:/logstash-6.1.1/data", "D:/logstash-6.1.1/lib", "D:/logstash-6.1.1/logs", "D:/logstash-6.1.1/logstash-core", "D:/logstash-6.1.1/logstash-core-plugin-api", "D:/logstash-6.1.1/modules", "D:/logstash-6.1.1/tools", "D:/logstash-6.1.1/vendor"]}
[2018-01-19T22:15:07,832][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"D:/logstash-6.1.1/logstash.conf"}
[2018-01-19T22:15:07,832][DEBUG][logstash.agent ] Converging pipelines
[2018-01-19T22:15:07,847][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x2a9aab53 sleep>"}
[2018-01-19T22:15:12,847][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x2a9aab53 sleep>"}
[2018-01-19T22:15:17,831][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["D:/logstash-6.1.1/CONTRIBUTORS", "D:/logstash-6.1.1/Gemfile", "D:/logstash-6.1.1/Gemfile.lock", "D:/logstash-6.1.1/LICENSE", "D:/logstash-6.1.1/NOTICE.TXT", "D:/logstash-6.1.1/bin", "D:/logstash-6.1.1/config", "D:/logstash-6.1.1/data", "D:/logstash-6.1.1/lib", "D:/logstash-6.1.1/logs", "D:/logstash-6.1.1/logstash-core", "D:/logstash-6.1.1/logstash-core-plugin-api", "D:/logstash-6.1.1/modules", "D:/logstash-6.1.1/tools", "D:/logstash-6.1.1/vendor"]}
[2018-01-19T22:15:17,831][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"D:/logstash-6.1.1/logstash.conf"}
[2018-01-19T22:15:17,831][DEBUG][logstash.agent ] Converging pipelines
[2018-01-19T22:15:17,847][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x2a9aab53 sleep>"}
[2018-01-19T22:15:22,862][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x2a9aab53 sleep>"}
I telnet in and that is the undefined ssl error came from, the filebeat server is where i ran telnet.


(Greg Sullivan) #6

from filebeat I got this,
ERR Failed to connect: remote error: tls: handshake failure


(Greg Sullivan) #7

Seems to be ok now, I added back the client certs and data started flowing.


(Pier-Hugues Pellerin) #8

You did not regenerate the certs?


(Greg Sullivan) #9

No I used the existing certs, I had a wrong path in filebeat for the drop folder. But after a restart logstash throwing different errors. I installed x-pack for monitoring.

Should I open another thread for the below?
I did not use IP's in the SSL, so don't know why it is trying to resolve them?

[2018-01-22T23:02:26,686][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<Manticore::UnknownException: Host name '10.210.0.129' does not match the certificate subject provided by the peer (CN=elastic1)>}
[2018-01-22T23:02:26,686][WARN ][logstash.outputs.elasticsearch] Error while performing sniffing {:error_message=>"Host name '10.210.0.129' does not match the certificate subject provided by the peer (CN=elastic1)", :class=>"Manticore::UnknownException", :backtrace=>["D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/response.rb:37:in block in initialize'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/response.rb:79:incall'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:74:in perform_request'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:287:inperform_request_to_url'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:274:in block in perform_request'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:369:inwith_connection'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in perform_request'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:163:incheck_sniff'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:156:in sniff!'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:145:inblock in start_sniffer'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:127:in until_stopped'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:143:inblock in start_sniffer'"]}
[2018-01-22T23:02:26,733][WARN ][logstash.licensechecker.licensereader] UNEXPECTED POOL ERROR {:e=>#<Manticore::UnknownException: Host name '10.210.0.129' does not match the certificate subject provided by the peer (CN=elastic1)>}
[2018-01-22T23:02:26,733][WARN ][logstash.licensechecker.licensereader] Error while performing sniffing {:error_message=>"Host name '10.210.0.129' does not match the certificate subject provided by the peer (CN=elastic1)", :class=>"Manticore::UnknownException", :backtrace=>["D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/response.rb:37:in block in initialize'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/response.rb:79:incall'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:74:in perform_request'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:287:inperform_request_to_url'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:274:in block in perform_request'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:369:inwith_connection'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in perform_request'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:163:incheck_sniff'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:156:in sniff!'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:145:inblock in start_sniffer'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:127:in until_stopped'", "D:/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:143:inblock in start_sniffer'"]}


(Greg Sullivan) #10

I figured the vm logstash is on needed replacement so I start over, did not install x-pack as that seems too cause to many issues.
After adding the CA cert to the Java keystore it seems to be working so I'm going to go without Logstash monitoring for now.
Thanks


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.