Fleet Agent: Logstash output: Invalid version of beats protocol

[Gustavo_Llermaly]

This is my Logstash config

input {
  elastic_agent {
    port => 5044
    ssl => false
  }
}

filter {
    mutate {
        add_field => {
            "foo" => "bar gustavito"
        }
    }
}

output {
    elasticsearch {
        cloud_auth => "xxxx:xxx"
        cloud_id => "my_cluster:..."
        data_stream => "true"
    }
    stdout {
        codec => rubydebug {
        }
    }

And getting back this:

[2022-09-17T19:18:01,567][WARN ][io.netty.channel.DefaultChannelPipeline][main][8a9c9f2dc6465fb8ed9835549cd253f9a62380bfff99a3dcf3bd608f17490322] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22

Fleet server is in Elastic Cloud

The agent lives in the same machine than Logstash, and the Logstash Output is set to localhost:5044 in the Fleet UI

Tried with certificates with no success (bad certificate error from Logstash) now, disabling ssl I get this error what I read is when the elasticsearch output is trying to send data to Logstash.

Any advice here?

Thank you

[ropc (Romain Chanu)]

@Gustavo_Llermaly - this error io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 indicates that there is a mismatch in protocol in the data received by the elastic_agent input plugin.

Can you clarify the following:

• Which Elastic Agent version are you using?
• Which Logstash version are you using?
• Which version of the elastic_agent input plugin are you using?

[Gustavo_Llermaly]

Hello @ropc

The entire stack is using 8.4.1.

This errors shows when I disable ssl . Is it possible to send data from Fleet agent to Logstash without certificates?

[ropc (Romain Chanu)]

Hi @Gustavo_Llermaly - let me do some research on that one. I am not aware of any explicit requirements when it comes to using Elastic Agent -> Logstash -> Integration Server in ESS.

[Gustavo_Llermaly]

Thank you!.

I was able to get it working specifying a dns (localhost) when creating the cert, and providing the same cert to Logstash and Fleet UI.

Ideally I want to use the same cert in all my Elastic agents (I will have 400+ agents)

[ropc (Romain Chanu)]

The connection between Logstash -> Integration Server (ESS) is encrypted - Elasticsearch Service uses standard publicly trusted certificates, so there’s no need specify other SSL settings in the Logstash pipeline.

You can refer to this documentation regarding the configuration of the Logstash pipeline: Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.4] | Elastic

[Gustavo_Llermaly]

Could you expand on "there's no need to specify other SSL settings in the Logstash pipeline" ?

The example looks like this:

  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/path/to/ca.crt"]
    ssl_certificate => "/path/to/logstash.crt"
    ssl_key => "/path/to/logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }

So I need a certificate and key for logstash, and a certificate and key for Fleet Logstash Output?

[ropc (Romain Chanu)]

Could you expand on "there's no need to specify other SSL settings in the Logstash pipeline" ?

I was referring to the Elasticsearch output section in Logstash.

This errors shows when I disable ssl . Is it possible to send data from Fleet agent to Logstash without certificates?

If you have installed a Fleet-managed Elastic Agent, it is impossible to save the Fleet Logstash output in the Fleet UI without adding the certificates that are used to communicate between Fleet <> Logstash (as per Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.4] | Elastic).

[Gustavo_Llermaly]

Thanks for your explanation. I'm clear the elasticsearch output doesnt need any certs.

My concern is the agent input, I need to do the simplest configuration. Which looks like it is a single cert with the logstash dns on it and shared between logstash and Fleet as disabling ssl is not possible.

Is this correct?

[ropc (Romain Chanu)]

That's my understanding as well - you will need to create the relevant certificates and use them in the configuration of the Logstash output for the Elastic Agent as well as the Logstash pipeline configuration.

[Nima_Rezainia]

Hi,

We enfroce mutual TLS between Agent and Logstash. You see this in the UI, when configuring the Logstash output.

[Gustavo_Llermaly]

I will summarize my weekend here, let me know If I'm making any imprecision.

1. TLS (ssl => true) is mandatory, if you set ssl to false you will receive bad protocol errors
2. You can create one certificate and use the same in Fleet, and in Logstash. The docs suggest to create 2: one for client (Fleet), one for server (Logstash). The Logstash one must have --ip or --dns set. The client one can omit those.
3. The logstash hostname (what you set in Fleet hosts) must match with what you configure under --dns or --ip in the certificate or you will receive "bad certificate" error <= this is the root cause of the error, I wasnt setting --dns