Hello,
I am currently trying to configure a logstash output for the fleet server. I followed the instruction from the documentation.
First I set up a Elasticsearch output for the fleet server. That worked fine. After that I configured Logstash as output which results in the problem that the connect fleet server stopped working... Setting the output back to Elastic did not work as well. I had to resetup the fleet server.
Then I took a deeper look into the logs, which was kind of odd. Because the logstash-plain.log showed me this:
[2023-01-26T14:28:52,294][INFO ][org.logstash.beats.BeatsHandler][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] [local: 172.22.52.167:5044, remote: 172.22.52.177:37882] Handling exception: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a (caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a)
[2023-01-26T14:28:52,294][WARN ][io.netty.channel.DefaultChannelPipeline][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
SNIP
The hex decoded message from the error looks like this (IP has been replaced):
GET / HTTP/1.1\r\nHost: my.ip.addr.167:5044\r\nUser-Agent: Elastic-Fleet-Server/8.6.0 (linux; amd64; 05088c13; 2023-01-04 19:26:24 +0000 UTC)\r\nX-Elastic-Client-Meta: es=7.16.0,go=1.18.9,t=7.16.0,hc=1.18.9\r\nX-Elastic-Product-Origin: fleet\r\nAccept-Encoding: gzip\r\n\r\n
Then I checked the fleet server, which told me that the connection of logstash was refused:
"Unit state changed fleet-server-default-fleet-server-fleet_server-7027ec44-f604-403c-bcf9-cb3c676f3651 (STARTING->FAILED): Error - read tcp [IP1]:38910->[IP2]:5044: read: connection reset by peer"
So it seems that the fleet server is sending plaintext (hex from error can be decoded to http request) to logstash. And logstash throws an exception, because it is expecting https.
How is this possible?
Here is my logstash pipeline:
input {
elastic_agent {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
ssl_certificate => "/etc/logstash/certs/logstash.crt"
ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
ssl_verify_mode => "force_peer"
}
}
output {
elasticsearch {
hosts => ["https://elastic1:9200", "https://elastic2:9200", "https://elastic3:9200", "https://elastic4:9200"]
api_key => "USER:PW"
data_stream => true
ssl => true
cacert => "/etc/logstash/certs/http_ca.crt"
}
}
And the certificate generation and setup for logstash was exactly like described here Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic
My Elastic Environment is in version 8.6.0
Can anyone help?