I have an elk setup (1 master ES, 3 worker es, 1 logstash, 1 kibana) with filebeat being the log collector/emitter. Post enabling x-pack and TLS, ES and Kibana is working fine. the problem is with logstash. I'm currently seeing this error in /var/log/logstash/logstash-plain.log.
[ERROR][logstash.javapipeline ][filebeat] Pipeline aborted due to error {:pipeline_id=>"filebeat", :exception=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'"
I can also see below logs in the elasticsearch master server:
[2020-09-01T07:13:20,323][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [esmasternode1] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.1.1.6:9300, remoteAddress=/publicipaddress:35166} [2020-09-01T07:13:20,865][WARN ][o.e.t.TcpTransport ] [esmasternode1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.1.6:9300, remoteAddress=/publicipaddress:35326}], closing connection
Below are my logstash and filebeat configurations. I have setup logstash as output in filebeat and filebeat as in input in my logstash config.
Logstash.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/logstash/logstashcert.crt"
ssl_key => "/etc/logstash/logstashcert.key"
}
}
filter { json { source => "message" remove_field => [ "message" ] } }
output {
elasticsearch {
hosts => ["https://esmasterprivateIP:9200"]
index => "logs-%{+YYYY-MM-dd}"
manage_template => true
template => "/etc/logstash/conf.d/template.json"
template_name => "mytemplate"
ssl => true
cacert => '/home/ubuntu/esca.pem'
user => logstash_user
password => mypassword
}
}
Filebeat.conf
output.logstash:
workers: 2
enabled: true
protocol: "https"
hosts: ['logstashprivateip:5044']
path: "/"
ssl:
certificate_authorities: [“/etc/tls.crt”]
I'm unable to trace down where I'm going wrong.
Note: Filebeat is running in kubernetes, Hence the config might look slightly different as it is passed through configmap.