Filebeats cannot push events to logstash

Elastic Stack Beats
1

Dear team,

could you please give me a hint what to do?
We have all Elasticstack on version 7.9.2 (ELS, Logstash, Filebeats, Kibana).
From filebeats towards logstash no TLS is used.

Here is the error found in logstash:

[2021-09-27T17:34:22,620][INFO ][org.logstash.beats.BeatsHandler][main][8d69d8d10a5180ce75149457893c4765bbb9f074270c3cef2aa4df5225038919] [local: 10.187.124.29:5044, remote: 10.187.124.25:47886] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: -12
[2021-09-27T17:34:22,621][WARN ][io.netty.channel.DefaultChannelPipeline][main][8d69d8d10a5180ce75149457893c4765bbb9f074270c3cef2aa4df5225038919] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: -12
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: -12
	at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.11.jar:?]
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.11.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
	... 11 more

Here is filbeat config:

# ============================== General ==============================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: "filebeat-kibana"

# ============================== Paths ==============================
path.home: "/app/global/filebeat/filebeat-7.9.2"
path.data: "/app/products/filebeat/data"
path.config: "/app/products/filebeat/conf"
path.logs: "/app/products/filebeat/logs"

# ============================== Filebeat inputs ==============================
filebeat.config.inputs:
  enabled: true
  path: /app/products/filebeat/conf/prospectors.d/*.yml

# ================================== Filebeat Outputs ===================================
# ------------------------------ Logstash Output -------------------------------

output.logstash:
    hosts: ["10.187.124.28:5044", "10.187.124.29:5044"]
    loadbalance: true

# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files:
    name: filebeat.log
    permissions: 0644
    keepfiles: 30
    interval: 24h
2

What is your logstash configuration ? Input -> Filter -> Output

3

There is no error for sure )

We have several other beats delivering to our system having different version.

input:

input {
  beats {
    port => 5044
    host => "ip_of_server"
  }
}

Filter pipelines we have many currently so I will not pass here.

4

Is it this single address generating the errors?

5

That is the address of filebeat instance which also has issues.
2021-09-27T17:30:43.123+0200 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 10.187.124.25:45350->10.187.124.29:5044: write: connection reset by peer

6

Can you share the filters and output? Alternatively, for a test, leave out the filters for now and see what errors are generated.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.