Fleet Server seems to sent plain HTTP to Fleets Logstash Output

m-logs · January 26, 2023, 3:30pm

Hello,

I am currently trying to configure a logstash output for the fleet server. I followed the instruction from the documentation.

First I set up a Elasticsearch output for the fleet server. That worked fine. After that I configured Logstash as output which results in the problem that the connect fleet server stopped working... Setting the output back to Elastic did not work as well. I had to resetup the fleet server.

Then I took a deeper look into the logs, which was kind of odd. Because the logstash-plain.log showed me this:

[2023-01-26T14:28:52,294][INFO ][org.logstash.beats.BeatsHandler][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] [local: 172.22.52.167:5044, remote: 172.22.52.177:37882] Handling exception: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a (caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a)
[2023-01-26T14:28:52,294][WARN ][io.netty.channel.DefaultChannelPipeline][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]

SNIP

The hex decoded message from the error looks like this (IP has been replaced):

GET / HTTP/1.1\r\nHost: my.ip.addr.167:5044\r\nUser-Agent: Elastic-Fleet-Server/8.6.0 (linux; amd64; 05088c13; 2023-01-04 19:26:24 +0000 UTC)\r\nX-Elastic-Client-Meta: es=7.16.0,go=1.18.9,t=7.16.0,hc=1.18.9\r\nX-Elastic-Product-Origin: fleet\r\nAccept-Encoding: gzip\r\n\r\n

Then I checked the fleet server, which told me that the connection of logstash was refused:

"Unit state changed fleet-server-default-fleet-server-fleet_server-7027ec44-f604-403c-bcf9-cb3c676f3651 (STARTING->FAILED): Error - read tcp [IP1]:38910->[IP2]:5044: read: connection reset by peer"

So it seems that the fleet server is sending plaintext (hex from error can be decoded to http request) to logstash. And logstash throws an exception, because it is expecting https.
How is this possible?

Here is my logstash pipeline:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    hosts => ["https://elastic1:9200", "https://elastic2:9200", "https://elastic3:9200", "https://elastic4:9200"]
    api_key => "USER:PW"
    data_stream => true
    ssl => true
    cacert => "/etc/logstash/certs/http_ca.crt"
  }
}

And the certificate generation and setup for logstash was exactly like described here Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic

My Elastic Environment is in version 8.6.0

Can anyone help?

m-logs · January 31, 2023, 9:49am

I updated elastic to version 8.6.1, but without success.

mverbeek · January 31, 2023, 11:38am

Hello,

Did you also configure SSL in the fleet server in the Logstash configuration there?

m-logs · January 31, 2023, 11:59am

Hello mverbeek,

Thanks for your reply.

I set up the certificates in the Fleet Settings, like described in the manual of elastic. Please see my screenshots.

The fleet servers are intiially enrolled (before switching via web GUI) in fleet mode like this:

elastic-agent enroll --url=https://fleet-1:8220 \
  --fleet-server-es=https://elastic-hot-1:9200 \
  --fleet-server-service-token=[MY_SERVICE_TOKEN] \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=[MY_TOKEN]\
  --certificate-authorities=/etc/elastic-agent/certs/http_ca.crt \
  --fleet-server-cert=/etc/elastic-agent/certs/fleet-1.crt \
  --fleet-server-cert-key=/etc/elastic-agent/certs/fleet-1.key

In the command line parameters of the elastic agent I only see "do-not-encrypt"-options, and not the opposite. This is why I think that the communication should actually be encrypted by default. Or do I miss something?

mverbeek · January 31, 2023, 12:27pm

So currently you are trying to configure the Elastic Agent to send data to Logstash, and then have Logstash forward it to the Elasticsearch cluster.

The management layer of Elastic Agent should not have a problem with this, or notice this at all.

Can you describe exactly what part stopped working?

Can you still enroll agents?, Does Kibana still list them as healthy? Have you made sure the SSL certificates are signed correctly?

m-logs · January 31, 2023, 12:54pm

Right.

As soon I switch the output from elastic nodes to logstash the fleet server show that the policy is outdated and the response timer waits until 5 minutes. After these 5 minutes the fleet servers are shown as offline.

In version 8.6.0 no logs are incoming anymore into the elastic cluster. After the update to 8.6.1 I see partial logs.
In 8.6.0 the status of the fleet servers also switched to "Unhealthy". In 8.6.1 it stays "Healthy" (for the mentioned 5 minutes).

I double checked the SSL certificates, which I derived from the autogenerated http CA of elastic. They are all valid and from the correct CA.

I will test the agent enrollment, after switching to Logstash-output.

m-logs · January 31, 2023, 1:08pm

And here a screenshot of the state after switching to logstash-output:

m-logs · January 31, 2023, 1:49pm

If I try to enroll an agent with Logstash as Fleet Output the enrollments fails with the following error message:

Error: fail to enroll: fail to execute request to fleet-server: EOF

nchaulet · January 31, 2023, 1:54pm

Hi @m-logs

Just to be sure are trying to use a logstash output for a Fleet Server agent policy?
This is a not supported use case and we are adding some restrictions in the future to prevent this to happen [Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub

m-logs · January 31, 2023, 2:10pm

I only have one output in the basic license... But when it is not possible to output fleet servers to, a different source (like the elastic cluster), it would never be possible to use Logstash with the basic license?

Then the sentence "Make sure your subscription level supports output to Logstash." finally makes sense from here Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic

m-logs · February 1, 2023, 1:51pm

I agree with the post [Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub.

It would also be helpful to state in the docs that logstash + fleet server + basic license is not possible to use in this combination. I was debugging this for a few days and will probably not be the only one.

DefensiveDepth · February 2, 2023, 11:27am

Yep, I also spent the last couple days troubleshooting this issue.

@nchaulet Can you please confirm that this is not a License issue, but a technical one, correct?

m-logs · February 2, 2023, 11:59am

The thing is with the basic licsense you can only define one output for all policies. Like this:

OUTPUT_1 = logstash <= this is the only output you have

Fleet Server Output = OUTPUT_1
Other Agents Output = OUTPUT_1

With the payed licenses you can define:

OUTPUT_1 = logstash
OUTPUT_2 = elastic

And you can set up your environment like this:

Fleet Server Output = OUTPUT_2 (elastic)
Other Agents Output = OUTPUT_1 (logstash)

And this is the only possible setup, because the fleet server cannot output to logstash.

These are your options with the basic licsense in the policy settings:

DefensiveDepth · February 14, 2023, 9:10pm

Asked for further clarification here: [Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub

DefensiveDepth · March 2, 2023, 4:13pm

This has been confirmed to be a bug:

github.com/elastic/kibana

[Fleet] Prevent output changing types from Elasticsearch -> Logstash if a Fleet Server policy uses it

opened 03:21PM - 27 Feb 23 UTC

kpollich

bug Team:Fleet

Fleet Server can _only_ output to Elasticsearch. We've made one key change in ht…tps://github.com/elastic/kibana/issues/148896 wherein a policy containing the Fleet Server integration cannot have its output set to a Logstash output. However, we need to ensure that the same validation applies when an existing output is edited to have its type changed from Elasticsearch to Output. If a Fleet Server policy exists that's pointing at an output, prevent its `type` attribute from being changed off of Elasticsearch. Users should be creating a separate output for their agents that output to Logstash, and this will prevent breaking their Fleet environment. ![image](https://user-images.githubusercontent.com/6766512/221604361-b067dcb4-5e74-4163-947b-312f87c4bf77.png) ![image](https://user-images.githubusercontent.com/6766512/221604385-826b1bb0-6552-4b98-9d92-766045a2777c.png)

system · March 30, 2023, 4:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fleet Agent: Logstash output: Invalid version of beats protocol Beats fleet	12	3041	October 17, 2022
No incoming data to Logstash Output from Elastic Agents - Only Elasticsearch ouptut works Elasticsearch fleet	6	657	May 28, 2023
No data streams and logs under fleet server Elastic Agent	1	335	March 14, 2023
Fleet server agent output not supported Elastic Agent fleet	2	285	August 22, 2023
Logstash Output for Fleet Managed Elastic Agent Elastic Agent	1	190	September 15, 2023

Fleet Server seems to sent plain HTTP to Fleets Logstash Output

Related topics