Beats to Logstash Connectivity

Hi Team

I am facing one weird issue when receiving connection on Logstash from filebeats.

Logstash Version : 7.7.0
Beats version: 7.7.0

Connection -

Beats -> Traefik LB -> Logstash Service -> Logstash

Configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
output.logstash:
  hosts: ["lb_name:443"]
  ssl.certificate_authorities: ["/opt/Kubernetes/test/filebeat-7.8.0-linux-x86_64/ca.crt"]
processors:
  - add_host_metadata: ~

Logstash Config

 input { beats { id => "node" port => "5044" }

 output { elasticsearch { id => "node" hosts => ["https://elastic.app:443"] index => "logstash" ssl => true ssl_certificate_verification => false user => "admin_user" password => "admin_user"} }

Error I can see in logs -

[2020-07-07T17:05:43,243][INFO ][org.logstash.beats.BeatsHandler][main][f3258ad2830e41205c470cb4d477e89218ab05a2156459fc45d75d590e13b54d] [local: 10.185.84.167:5044, remote: 10.185.84.58:46532] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 
[2020-07-07T17:05:43,245][WARN ][io.netty.channel.DefaultChannelPipeline][main][f3258ad2830e41205c470cb4d477e89218ab05a2156459fc45d75d590e13b54d] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. 
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final] 
at java.lang.Thread.run(Thread.java:834) [?:?] 
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 
at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.9.jar:?] 
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.9.jar:?] 
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
... 8 more 
[2020-07-07T17:05:43,329][INFO ][org.logstash.beats.BeatsHandler][main][f3258ad2830e41205c470cb4d477e89218ab05a2156459fc45d75d590e13b54d] [local: 10.185.84.167:5044, remote: 10.185.84.58:46532] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3 
[2020-07-07T17:05:43,335][WARN ][io.netty.channel.DefaultChannelPipeline][main][f3258ad2830e41205c470cb4d477e89218ab05a2156459fc45d75d590e13b54d] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. 
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3 
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final] 
at java.lang.Thread.run(Thread.java:834) [?:?] 
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3 
at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.9.jar:?] 
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.9.jar:?] 
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] 
... 10 more

Any insights will be helpful. Thanks.

Something is connecting to logstash that is not speaking the lumberjack protocol. Does the LB do a health check?

Yes, LB performs a health check.

If the health check consists of more than a check that the port is listening (i.e. it does some kind of ECV) then it would result in that error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.