# Fleet Server seems to sent plain HTTP to Fleets Logstash Output **URL:** \ **Category:** Elastic Agent\ **Tags:** fleet\ **Created:** [January 26, 2023, 3:30pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035 "2023-01-26T15:30:44Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 26, 2023, 3:30pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/1 "2023-01-26T15:30:44Z")
Hello, I am currently trying to configure a logstash output for the fleet server. I followed the instruction from the documentation. First I set up a Elasticsearch output for the fleet server. That worked fine. After that I configured Logstash as output which results in the problem that the connect fleet server stopped working... Setting the output back to Elastic did not work as well. I had to resetup the fleet server. Then I took a deeper look into the logs, which was kind of odd. Because the logstash-plain.log showed me this: ```auto [2023-01-26T14:28:52,294][INFO][org.logstash.beats.BeatsHandler][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] [local: 172.22.52.167:5044, remote: 172.22.52.177:37882] Handling exception: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a (caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a) [2023-01-26T14:28:52,294][WARN][io.netty.channel.DefaultChannelPipeline][elastic-agent-pipeline][91ba581278197c7a2484b72e95f5f4d262b91e58dff384225ffbaf61a266e49c] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206d792e69702e616464722e3136373a353034340d0a557365722d4167656e743a20456c61737469632d466c6565742d5365727665722f382e362e3020286c696e75783b20616d6436343b2030353038386331333b20323032332d30312d30342031393a32363a3234202b3030303020555443290d0a582d456c61737469632d436c69656e742d4d6574613a2065733d372e31362e302c676f3d312e31382e392c743d372e31362e302c68633d312e31382e390d0a582d456c61737469632d50726f647563742d4f726967696e3a20666c6565740d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] SNIP ``` The hex decoded message from the error looks like this (IP has been replaced): ```auto GET / HTTP/1.1\r\nHost: my.ip.addr.167:5044\r\nUser-Agent: Elastic-Fleet-Server/8.6.0 (linux; amd64; 05088c13; 2023-01-04 19:26:24 +0000 UTC)\r\nX-Elastic-Client-Meta: es=7.16.0,go=1.18.9,t=7.16.0,hc=1.18.9\r\nX-Elastic-Product-Origin: fleet\r\nAccept-Encoding: gzip\r\n\r\n ``` Then I checked the fleet server, which told me that the connection of logstash was refused: ```auto "Unit state changed fleet-server-default-fleet-server-fleet_server-7027ec44-f604-403c-bcf9-cb3c676f3651 (STARTING->FAILED): Error - read tcp [IP1]:38910->[IP2]:5044: read: connection reset by peer" ``` So it seems that the fleet server is sending plaintext (hex from error can be decoded to http request) to logstash. And logstash throws an exception, because it is expecting https. How is this possible? Here is my logstash pipeline: ```auto input { elastic_agent { port => 5044 ssl => true ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"] ssl_certificate => "/etc/logstash/certs/logstash.crt" ssl_key => "/etc/logstash/certs/logstash.pkcs8.key" ssl_verify_mode => "force_peer" } } output { elasticsearch { hosts => ["https://elastic1:9200", "https://elastic2:9200", "https://elastic3:9200", "https://elastic4:9200"] api_key => "USER:PW" data_stream => true ssl => true cacert => "/etc/logstash/certs/http_ca.crt" } } ``` And the certificate generation and setup for logstash was exactly like described here [Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic](https://www.elastic.co/guide/en/fleet/8.6/secure-logstash-connections.html) My Elastic Environment is in version 8.6.0 Can anyone help? ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 31, 2023, 9:49am UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/2 "2023-01-31T09:49:24Z")
I updated elastic to version 8.6.1, but without success. ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [January 31, 2023, 11:38am UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/3 "2023-01-31T11:38:13Z")
Hello, Did you also configure SSL in the fleet server in the Logstash configuration there? ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 31, 2023, 11:59am UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/4 "2023-01-31T11:59:29Z")
Hello mverbeek, Thanks for your reply. I set up the certificates in the Fleet Settings, like described in the manual of elastic. Please see my screenshots. ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a3f25f73c037239d80727473c69944748744db2.png) ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a835c10250d73e9ebc4f02ffc1b3319f81acb42.png) The fleet servers are intiially enrolled (before switching via web GUI) in fleet mode like this: ```auto elastic-agent enroll --url=https://fleet-1:8220 \ --fleet-server-es=https://elastic-hot-1:9200 \ --fleet-server-service-token=[MY_SERVICE_TOKEN] \ --fleet-server-policy=fleet-server-policy \ --fleet-server-es-ca-trusted-fingerprint=[MY_TOKEN]\ --certificate-authorities=/etc/elastic-agent/certs/http_ca.crt \ --fleet-server-cert=/etc/elastic-agent/certs/fleet-1.crt \ --fleet-server-cert-key=/etc/elastic-agent/certs/fleet-1.key ``` In the command line parameters of the elastic agent I only see "do-not-encrypt"-options, and not the opposite. This is why I think that the communication should actually be encrypted by default. Or do I miss something? ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [January 31, 2023, 12:27pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/5 "2023-01-31T12:27:32Z")
So currently you are trying to configure the Elastic Agent to send data to Logstash, and then have Logstash forward it to the Elasticsearch cluster. The management layer of Elastic Agent should not have a problem with this, or notice this at all. Can you describe exactly what part stopped working? Can you still enroll agents?, Does Kibana still list them as healthy? Have you made sure the SSL certificates are signed correctly? ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 31, 2023, 12:54pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/6 "2023-01-31T12:54:58Z")
Right. As soon I switch the output from elastic nodes to logstash the fleet server show that the policy is outdated and the response timer waits until 5 minutes. After these 5 minutes the fleet servers are shown as offline. In version 8.6.0 no logs are incoming anymore into the elastic cluster. After the update to 8.6.1 I see partial logs. In 8.6.0 the status of the fleet servers also switched to "Unhealthy". In 8.6.1 it stays "Healthy" (for the mentioned 5 minutes). I double checked the SSL certificates, which I derived from the autogenerated http CA of elastic. They are all valid and from the correct CA. I will test the agent enrollment, after switching to Logstash-output. ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 31, 2023, 1:08pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/7 "2023-01-31T13:08:29Z")
And here a screenshot of the state after switching to logstash-output: ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2fd4a572872b13d6b07b60ab4d3e31b22b8d64cf.png) ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 31, 2023, 1:49pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/8 "2023-01-31T13:49:55Z")
If I try to enroll an agent with Logstash as Fleet Output the enrollments fails with the following error message: ```auto Error: fail to enroll: fail to execute request to fleet-server: EOF ``` ---
**Author:** ![nchaulet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchaulet/32/77896_2.png) [@nchaulet](https://discuss.elastic.co/u/nchaulet)\ **Post date:** [January 31, 2023, 1:54pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/9 "2023-01-31T13:54:32Z")
Hi @m-logs Just to be sure are trying to use a logstash output for a Fleet Server agent policy? This is a not supported use case and we are adding some restrictions in the future to prevent this to happen [[Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/148896) ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [January 31, 2023, 2:10pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/10 "2023-01-31T14:10:42Z")
I only have one output in the basic license... But when it is not possible to output fleet servers to, a different source (like the elastic cluster), it would never be possible to use Logstash with the basic license? Then the sentence "Make sure your subscription level supports output to Logstash." finally makes sense from here [Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic](https://www.elastic.co/guide/en/fleet/current/secure-logstash-connections.html) ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [February 1, 2023, 1:51pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/11 "2023-02-01T13:51:25Z")
I agree with the post [[Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/148896). It would also be helpful to state in the docs that logstash + fleet server + basic license is not possible to use in this combination. I was debugging this for a few days and will probably not be the only one. ---
**Author:** ![DefensiveDepth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defensivedepth/32/110034_2.png) [@DefensiveDepth](https://discuss.elastic.co/u/DefensiveDepth)\ **Post date:** [February 2, 2023, 11:27am UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/12 "2023-02-02T11:27:00Z")
Yep, I also spent the last couple days troubleshooting this issue. @nchaulet Can you please confirm that this is not a License issue, but a technical one, correct? ---
**Author:** ![m-logs](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@m-logs](https://discuss.elastic.co/u/m-logs)\ **Post date:** [February 2, 2023, 11:59am UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/13 "2023-02-02T11:59:27Z")
The thing is with the basic licsense you can only define one output for all policies. Like this: OUTPUT\_1 = logstash \<= this is the only output you have Fleet Server Output = OUTPUT\_1 Other Agents Output = OUTPUT\_1 With the payed licenses you can define: OUTPUT\_1 = logstash OUTPUT\_2 = elastic And you can set up your environment like this: Fleet Server Output = OUTPUT\_2 (elastic) Other Agents Output = OUTPUT\_1 (logstash) And this is the only possible setup, because the fleet server cannot output to logstash. These are your options with the basic licsense in the policy settings: ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/5/9/59b5bc28d061c5b8ac1c2ac2700d3ec96dc67333.png) ---
**Author:** ![DefensiveDepth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defensivedepth/32/110034_2.png) [@DefensiveDepth](https://discuss.elastic.co/u/DefensiveDepth)\ **Post date:** [February 14, 2023, 9:10pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/14 "2023-02-14T21:10:22Z")
Asked for further clarification here: [[Fleet] Fleet Server Policy should forbid setting Logstash as its output · Issue #148896 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/148896#issuecomment-1430381946) ---
**Author:** ![DefensiveDepth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defensivedepth/32/110034_2.png) [@DefensiveDepth](https://discuss.elastic.co/u/DefensiveDepth)\ **Post date:** [March 2, 2023, 4:13pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/15 "2023-03-02T16:13:53Z")
This has been confirmed to be a bug: > > > Fleet Server can \_only\_ output to Elasticsearch. We've made one key change in ht…tps://github.com/elastic/kibana/issues/148896 wherein a policy containing the Fleet Server integration cannot have its output set to a Logstash output. However, we need to ensure that the same validation applies when an existing output is edited to have its type changed from Elasticsearch to Output. > > If a Fleet Server policy exists that's pointing at an output, prevent its \`type\` attribute from being changed off of Elasticsearch. Users should be creating a separate output for their agents that output to Logstash, and this will prevent breaking their Fleet environment. > > !\[image\](https://user-images.githubusercontent.com/6766512/221604361-b067dcb4-5e74-4163-947b-312f87c4bf77.png) > > !\[image\](https://user-images.githubusercontent.com/6766512/221604385-826b1bb0-6552-4b98-9d92-766045a2777c.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 30, 2023, 4:14pm UTC](https://discuss.elastic.co/t/fleet-server-seems-to-sent-plain-http-to-fleets-logstash-output/324035/16 "2023-03-30T16:14:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.