Invalid protocol when filebeat send to logstash

Elastic Stack Beats
1 
[2022-10-15T11:01:12,651][INFO ][org.logstash.beats.BeatsHandler][terza][7bd5f0adbbfa46b9e9ef1e064135c6f8c54b821b69f260da353a735193b5fe7d] [local: 172.19.1.12:5044, remote: 172.19.0.1:53380] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71)
[2022-10-15T11:01:12,655][WARN ][io.netty.channel.DefaultChannelPipeline][terza][7bd5f0adbbfa46b9e9ef1e064135c6f8c54b821b69f260da353a735193b5fe7d] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

hello i use filebeat 7 and logstash 8 is possible?

2

According to the support matrix it seems Beats 7.17 is the earliest version compatible with Logstash 8.

3

hell this is my conf on logstash

input {
      beats {  # Notice the input is now being taken from 'Beats' instead of a 'file'
        type => "logs"
        port => "5044" 
      }
}



output {

		elasticsearch {
			hosts => ["elasticsearch:9200"]
			index => "prova"
		}
		

		stdout { codec => rubydebug }
}
4

Which version of Filebeat are you using?

5

now i use 8.4.3

i change config but

[2022-10-15T11:53:19,093][ERROR][logstash.inputs.tcp      ][terza][29b59b8e3f0e92a4cd1027566c11d7d30488054b6d1fb1a8ce632fd78656b0a1] localhost/127.0.0.1:60412: closing due:
org.jruby.exceptions.RuntimeError: (RuntimeError) Not implemented


input {
    tcp {
        port => 5044
        type => logs
        codec => rubydebug
    }
}


output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "prova2"
		}
		

		stdout { codec => rubydebug }
}
6

Try removing this. A rubydebug codec does not make sense for an input.

7

if use in logstash conf

input {
    beats {
      host => "localhost"
      port => 5044
    }
}


output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "prova2"
		}
		

		stdout { codec => rubydebug }
}

[2022-10-15T12:16:04,651][INFO ][org.logstash.beats.BeatsHandler][terza][0bc359621600c3614b91a3cd92e6ee0d262ea0ed4fe2b06cfcd2d59f46baf37f] [local: 127.0.0.1:5044, remote: 127.0.0.1:45118] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69)
[2022-10-15T12:16:04,651][WARN ][io.netty.channel.DefaultChannelPipeline][terza][0bc359621600c3614b91a3cd92e6ee0d262ea0ed4fe2b06cfcd2d59f46baf37f] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
	at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.4.1.jar:?]
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.4.1.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	... 11 more
8

filebeat conf

filebeat.inputs:
  - type: log
    enabled: true
    paths:
        - \FSEBroker\*.*
      

output.elasticsearch:
  hosts: [ "127.0.0.1:5044"]
  protocol: "http"
  #index: "fss2-%{+yyyy.MM.d
9

This is for sending it to Elasticsearch, which you are not doing. Instead configure it to send data to Logstash.

10

sorry but if i change the config with

input {
    tcp {
        port => 5000
        type => syslog
        codec => json_lines
    }
}

in this case work but file log is not json in this case

11

difference in logstash log with config tcp

[2022-10-15T12:29:20,826][INFO ][logstash.inputs.tcp ][terza][02b3dac46b0b3d31eabbcf551e1292d22db8816cf5436cbcac98af1463933661] Starting tcp input listener {:address=>"0.0.0.0:5044", :ssl_enable=>false}

and

[2022-10-15T12:15:06,822][INFO ][org.logstash.beats.Server][terza][0bc359621600c3614b91a3cd92e6ee0d262ea0ed4fe2b06cfcd2d59f46baf37f] Starting server on port: 5044

12

if use tcp index is 8k but original file send is 20mega

input {
    tcp {
        port => 5044
        #type => syslog
        #codec => json_lines
    }
}

image
image1585Ă—56 3.44 KB

13

You should use the beats input and configure your filebeat to use the logstash output, anything else will not work as expected.

Beats use a custom protocol over tcp, so a TCP input will not work for beats messages.

You need to have only this output in your filebeat:

output.logstash:
  hosts: [ "127.0.0.1:5044"]

And you need to have only this input in your configuration:

input {
    beats {
      host => "localhost"
      port => 5044
    }
}
1 Like
14

yes sorry is different

15

is possible use multiline config?

input {
    beats {
      host => "localhost"
      port => 5044
    }
    
    codec => multiline {
              pattern => "^%{TIMESTAMP_ISO8601} "
              negate => true
              what => "previous"
        }
}

output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "fss_1"
		}
		

		stdout { codec => rubydebug }
}
16

If you need multiline you need to configure it in Filebeat, not Logstash, configuring multiline in Logstash for messages sent by filebeat will not work.

Check the documentation

17

ok tnx very much

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.