Invalid protocol when filebeat send to logstash

[Nikolas1306 (Nikolas “Nikolas”)]

[2022-10-15T11:01:12,651][INFO ][org.logstash.beats.BeatsHandler][terza][7bd5f0adbbfa46b9e9ef1e064135c6f8c54b821b69f260da353a735193b5fe7d] [local: 172.19.1.12:5044, remote: 172.19.0.1:53380] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71)
[2022-10-15T11:01:12,655][WARN ][io.netty.channel.DefaultChannelPipeline][terza][7bd5f0adbbfa46b9e9ef1e064135c6f8c54b821b69f260da353a735193b5fe7d] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

hello i use filebeat 7 and logstash 8 is possible?

[Christian_Dahlqvist]

According to the support matrix it seems Beats 7.17 is the earliest version compatible with Logstash 8.

[Nikolas1306 (Nikolas “Nikolas”)]

hell this is my conf on logstash

input {
      beats {  # Notice the input is now being taken from 'Beats' instead of a 'file'
        type => "logs"
        port => "5044" 
      }
}



output {

		elasticsearch {
			hosts => ["elasticsearch:9200"]
			index => "prova"
		}
		

		stdout { codec => rubydebug }
}

[Christian_Dahlqvist]

Which version of Filebeat are you using?

[Nikolas1306 (Nikolas “Nikolas”)]

now i use 8.4.3

i change config but

[2022-10-15T11:53:19,093][ERROR][logstash.inputs.tcp      ][terza][29b59b8e3f0e92a4cd1027566c11d7d30488054b6d1fb1a8ce632fd78656b0a1] localhost/127.0.0.1:60412: closing due:
org.jruby.exceptions.RuntimeError: (RuntimeError) Not implemented

input {
    tcp {
        port => 5044
        type => logs
        codec => rubydebug
    }
}


output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "prova2"
		}
		

		stdout { codec => rubydebug }
}

[Christian_Dahlqvist]

@Nikolas1306

type => logs
codec => rubydebug

Try removing this. A rubydebug codec does not make sense for an input.

[Nikolas1306 (Nikolas “Nikolas”)]

if use in logstash conf

input {
    beats {
      host => "localhost"
      port => 5044
    }
}


output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "prova2"
		}
		

		stdout { codec => rubydebug }
}

[2022-10-15T12:16:04,651][INFO ][org.logstash.beats.BeatsHandler][terza][0bc359621600c3614b91a3cd92e6ee0d262ea0ed4fe2b06cfcd2d59f46baf37f] [local: 127.0.0.1:5044, remote: 127.0.0.1:45118] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69)
[2022-10-15T12:16:04,651][WARN ][io.netty.channel.DefaultChannelPipeline][terza][0bc359621600c3614b91a3cd92e6ee0d262ea0ed4fe2b06cfcd2d59f46baf37f] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
	at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.4.1.jar:?]
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.4.1.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	... 11 more

[Nikolas1306 (Nikolas “Nikolas”)]

filebeat conf

filebeat.inputs:
  - type: log
    enabled: true
    paths:
        - \FSEBroker\*.*
      

output.elasticsearch:
  hosts: [ "127.0.0.1:5044"]
  protocol: "http"
  #index: "fss2-%{+yyyy.MM.d

[Christian_Dahlqvist]

@Nikolas1306

output.elasticsearch:
  hosts: [ "127.0.0.1:5044"]
  protocol: "http"

This is for sending it to Elasticsearch, which you are not doing. Instead configure it to send data to Logstash.

[Nikolas1306 (Nikolas “Nikolas”)]

sorry but if i change the config with

input {
    tcp {
        port => 5000
        type => syslog
        codec => json_lines
    }
}

in this case work but file log is not json in this case

[Nikolas1306 (Nikolas “Nikolas”)]

difference in logstash log with config tcp

[2022-10-15T12:29:20,826][INFO ][logstash.inputs.tcp ][terza][02b3dac46b0b3d31eabbcf551e1292d22db8816cf5436cbcac98af1463933661] Starting tcp input listener {:address=>"0.0.0.0:5044", :ssl_enable=>false}

and

[2022-10-15T12:15:06,822][INFO ][org.logstash.beats.Server][terza][0bc359621600c3614b91a3cd92e6ee0d262ea0ed4fe2b06cfcd2d59f46baf37f] Starting server on port: 5044

[Nikolas1306 (Nikolas “Nikolas”)]

if use tcp index is 8k but original file send is 20mega

input {
    tcp {
        port => 5044
        #type => syslog
        #codec => json_lines
    }
}

[leandrojmp (Leandro Pereira)]

You should use the beats input and configure your filebeat to use the logstash output, anything else will not work as expected.

Beats use a custom protocol over tcp, so a TCP input will not work for beats messages.

You need to have only this output in your filebeat:

output.logstash:
  hosts: [ "127.0.0.1:5044"]

And you need to have only this input in your configuration:

input {
    beats {
      host => "localhost"
      port => 5044
    }
}

[Nikolas1306 (Nikolas “Nikolas”)]

@leandrojmp

input {
    beats {
      host => "localhost"
      port => 5044
    }
}

yes sorry is different

[Nikolas1306 (Nikolas “Nikolas”)]

is possible use multiline config?

input {
    beats {
      host => "localhost"
      port => 5044
    }
    
    codec => multiline {
              pattern => "^%{TIMESTAMP_ISO8601} "
              negate => true
              what => "previous"
        }
}

output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "fss_1"
		}
		

		stdout { codec => rubydebug }
}

[leandrojmp (Leandro Pereira)]

If you need multiline you need to configure it in Filebeat, not Logstash, configuring multiline in Logstash for messages sent by filebeat will not work.

Check the documentation

[Nikolas1306 (Nikolas “Nikolas”)]

@leandrojmp

Se hai bisogno di multiline devi configurarlo in Filebeat, non logstash, la configurazione multilinea in Logstash per i messaggi inviati da filebeat non funzionerà.

ok tnx very much