Hello there:
I set up the SSL /TLS on logstash/elasticsearch and kibnana as indicated above.
Everything looks fine, all are up and running, and I can see the beats(file/metric/etc...) are sending data on Kibana (Discover) via logstash to elasticseach nodes.
However, I still can see the following errors in the logstash-plain.log. It complains all elastic nodes but beats pipeline seems working fine. And all the nodes (Elasticsearch nodes, logstash, kibana, anc logstash pipeline config files) are setup with SSL/TLS, except on the clients (where beats were installed and fetching data from). I checked the doc here: https://www.elastic.co/guide/en/elastic-stack-overview/current/trb-security-ssl.html
[2018-10-09T10:15:33,110][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport]
[elasticnod1] caught exception while handling client http traffic, closing connection
[id: 0x36a4af1f, L:0.0.0.0/0.0.0.0:9200 ! R:/10.100.12.11:41864]io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a20656c61737469637365617263682e686c732e6478632e636f6d..
...............
Here is the logstash.yml
============
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.elasticsearch.url: ["https://elasticnode1:9200", "https://elasticnode2:9200" ]
xpack.monitoring.elasticsearch.ssl.truststore.path: "/etc/logstash/elastic-certificates.p12"
xpack.monitoring.elasticsearch.ssl.truststore.password: password
xpack.monitoring.elasticsearch.ssl.keystore.path: "/etc/logstash/elastic-certificates.p12"
xpack.monitoring.elasticsearch.ssl.keystore.password: password
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 60s
xpack.monitoring.collection.pipeline.details.enabled: true
====================
Here is the elasticsearch config on both elasticsearch nodes (each node has their own elastic-certificates.p12 corresponding to their own hostnames)
===================
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.monitoring.collection.interval: 60s
xpack.monitoring.collection.cluster.stats.timeout: 60s
xpack.monitoring.history.duration: 90d
xpack.watcher.history.cleaner_service.enabled: true
xpack.http.proxy.host: 'proxy host'
xpack.http.proxy.port: 3128
xpack.watcher.enabled: true
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/keys/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/keys/elastic-certificates.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/keys/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/keys/elastic-certificates.p1
===================================
here is the beats pipeline config (beat-pipeline.conf):
=================
input {
beats {
port => 5044
client_inactivity_timeout => 120
#ssl => false
}
}
output {
elasticsearch {
user => "logstash_ingest"
password => "password"
ssl => true
ssl_certificate_verification => true
cacert => "/etc/logstash/elastic-stack-ca.pem"
action => "index"
hosts => ["elactisnode1", "elasticnode2"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Please help and see if there is anything missing or incorrect, help is needed here, indeed.
Thanks a lot
Li