Fleet Server seems to sent plain HTTP to Fleets Logstash Output

Hello,

I am currently trying to configure a logstash output for the fleet server. I followed the instruction from the documentation.

First I set up a Elasticsearch output for the fleet server. That worked fine. After that I configured Logstash as output which results in the problem that the connect fleet server stopped working... Setting the output back to Elastic did not work as well. I had to resetup the fleet server.

Then I took a deeper look into the logs, which was kind of odd. Because the logstash-plain.log showed me this:

[2023-01-26T14:28:52,294][INFO ][org.logstash.beats.BeatsHandler][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] [local: 172.22.52.167:5044, remote: 172.22.52.177:37882] Handling exception: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a (caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a)
[2023-01-26T14:28:52,294][WARN ][io.netty.channel.DefaultChannelPipeline][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]

SNIP

The hex decoded message from the error looks like this (IP has been replaced):

GET / HTTP/1.1\r\nHost: my.ip.addr.167:5044\r\nUser-Agent: Elastic-Fleet-Server/8.6.0 (linux; amd64; 05088c13; 2023-01-04 19:26:24 +0000 UTC)\r\nX-Elastic-Client-Meta: es=7.16.0,go=1.18.9,t=7.16.0,hc=1.18.9\r\nX-Elastic-Product-Origin: fleet\r\nAccept-Encoding: gzip\r\n\r\n

Then I checked the fleet server, which told me that the connection of logstash was refused:

"Unit state changed fleet-server-default-fleet-server-fleet_server-7027ec44-f604-403c-bcf9-cb3c676f3651 (STARTING->FAILED): Error - read tcp [IP1]:38910->[IP2]:5044: read: connection reset by peer"

So it seems that the fleet server is sending plaintext (hex from error can be decoded to http request) to logstash. And logstash throws an exception, because it is expecting https.
How is this possible?

Here is my logstash pipeline:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    hosts => ["https://elastic1:9200", "https://elastic2:9200", "https://elastic3:9200", "https://elastic4:9200"]
    api_key => "USER:PW"
    data_stream => true
    ssl => true
    cacert => "/etc/logstash/certs/http_ca.crt"
  }
}

And the certificate generation and setup for logstash was exactly like described here Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic

My Elastic Environment is in version 8.6.0

Can anyone help?

I updated elastic to version 8.6.1, but without success.

Hello,

Did you also configure SSL in the fleet server in the Logstash configuration there?

Hello mverbeek,

Thanks for your reply.

I set up the certificates in the Fleet Settings, like described in the manual of elastic. Please see my screenshots.

The fleet servers are intiially enrolled (before switching via web GUI) in fleet mode like this:

elastic-agent enroll --url=https://fleet-1:8220 \
  --fleet-server-es=https://elastic-hot-1:9200 \
  --fleet-server-service-token=[MY_SERVICE_TOKEN] \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=[MY_TOKEN]\
  --certificate-authorities=/etc/elastic-agent/certs/http_ca.crt \
  --fleet-server-cert=/etc/elastic-agent/certs/fleet-1.crt \
  --fleet-server-cert-key=/etc/elastic-agent/certs/fleet-1.key

In the command line parameters of the elastic agent I only see "do-not-encrypt"-options, and not the opposite. This is why I think that the communication should actually be encrypted by default. Or do I miss something?

So currently you are trying to configure the Elastic Agent to send data to Logstash, and then have Logstash forward it to the Elasticsearch cluster.

The management layer of Elastic Agent should not have a problem with this, or notice this at all.

Can you describe exactly what part stopped working?

Can you still enroll agents?, Does Kibana still list them as healthy? Have you made sure the SSL certificates are signed correctly?

Right.

As soon I switch the output from elastic nodes to logstash the fleet server show that the policy is outdated and the response timer waits until 5 minutes. After these 5 minutes the fleet servers are shown as offline.

In version 8.6.0 no logs are incoming anymore into the elastic cluster. After the update to 8.6.1 I see partial logs.
In 8.6.0 the status of the fleet servers also switched to "Unhealthy". In 8.6.1 it stays "Healthy" (for the mentioned 5 minutes).

I double checked the SSL certificates, which I derived from the autogenerated http CA of elastic. They are all valid and from the correct CA.

I will test the agent enrollment, after switching to Logstash-output.

And here a screenshot of the state after switching to logstash-output:

If I try to enroll an agent with Logstash as Fleet Output the enrollments fails with the following error message:

Error: fail to enroll: fail to execute request to fleet-server: EOF

Hi @m-logs

Just to be sure are trying to use a logstash output for a Fleet Server agent policy?
This is a not supported use case and we are adding some restrictions in the future to prevent this to happen [Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub

2 Likes

I only have one output in the basic license... But when it is not possible to output fleet servers to, a different source (like the elastic cluster), it would never be possible to use Logstash with the basic license?

Then the sentence "Make sure your subscription level supports output to Logstash." finally makes sense from here Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic

I agree with the post [Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub.

It would also be helpful to state in the docs that logstash + fleet server + basic license is not possible to use in this combination. I was debugging this for a few days and will probably not be the only one.

1 Like

Yep, I also spent the last couple days troubleshooting this issue.

@nchaulet Can you please confirm that this is not a License issue, but a technical one, correct?

The thing is with the basic licsense you can only define one output for all policies. Like this:

OUTPUT_1 = logstash <= this is the only output you have

Fleet Server Output = OUTPUT_1
Other Agents Output = OUTPUT_1

With the payed licenses you can define:

OUTPUT_1 = logstash
OUTPUT_2 = elastic

And you can set up your environment like this:

Fleet Server Output = OUTPUT_2 (elastic)
Other Agents Output = OUTPUT_1 (logstash)

And this is the only possible setup, because the fleet server cannot output to logstash.

These are your options with the basic licsense in the policy settings:

Asked for further clarification here: [Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub

This has been confirmed to be a bug:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.