InvalidFrameProtocolException and Connection reset by peer error

Eva · May 20, 2019, 7:30am

Hi there

I have been trying to use Filebeat to push the logs to logstash and then output it to AWS ES domain. The logs are shown up on AWS Elasticsearch domain when I do not use SSL authentication between Filebeat and Logstash. But when i try to use SSL mutual authentication [as mentioned here : https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step] between Logstash and Filebeat I see below errors:

Errors logged on Logstash :

[2019-05-20T06:56:04,209][INFO ][org.logstash.beats.BeatsHandler] [local: ip:5044, remote: ip:58800] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
[2019-05-20T06:56:04,209][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 10 more
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 8 more

======================================================

Errors seen on Filebeat :

2019-05-19T07:30:29Z ERR Connecting error publishing events (retrying): read tcp ip:55994->ip:5044: read: connection reset by peer
2019-05-19T07:30:30Z ERR Connecting error publishing events (retrying): read tcp ip:55996->ip:5044: read: connection reset by peer
2019-05-19T07:30:32Z ERR Connecting error publishing events (retrying): read tcp ip:55998->ip:5044: read: connection reset by peer
2019-05-19T07:30:37Z ERR Connecting error publishing events (retrying): read tcp ip:56000->ip:5044: read: connection reset by peer
2019-05-19T07:30:45Z ERR Connecting error publishing events (retrying): read tcp ip:56002->ip:5044: read: connection reset by peer

2019-05-20T06:57:40Z INFO No non-zero metrics in the last 30s
2019-05-20T06:58:05Z ERR Connecting error publishing events (retrying): read tcp ip:58804->ip:5044: read: connection reset by peer
2019-05-20T06:58:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=155
2019-05-20T06:58:40Z INFO No non-zero metrics in the last 30s
2019-05-20T06:59:06Z ERR Connecting error publishing events (retrying): read tcp ip:58806->ip:5044: read: connection reset by peer
2019-05-20T06:59:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=155
2019-05-20T06:59:40Z INFO No non-zero metrics in the last 30s

Here is how my Filebeat config look like:
filebeat.prospectors:

paths:
- /var/log/*.log
  fields:
  application: GS
  account: 1y
  env: prodtest
  name: "GSprod"

output.logstash:
template.name: "filebeat"
template.path: "filebeat.template.json"
hosts: ["www.mydomain.com:5044"]
ssl.certificate_authorities: ["/ssl/ca.crt"]
ssl.certificate: "/sslbeat.crt"
ssl.key: "/beat.key"

Here is how my Logstash config is(removed the filter section as its huge):

input {
beats {
port => 5044
}
beats {
port => 5045
client_inactivity_timeout => 120
ssl => true
ssl_certificate_authorities => ["/opt/ssl/ca.crt"]
ssl_certificate => "/opt/ssl/logstash.crt"
ssl_key => "/opt/ssl/logstash.key"
ssl_verify_mode => "force_peer"
}

}

output {
amazon_es {
hosts => ["vpc-------------es.amazonaws.com"]
region => "Region"
aws_access_key_id => "AK"
aws_secret_access_key => "SK"
index => "logstash-%{+YYYY.MM.dd}"
}
}

Logstash version - logstash 6.7.2
Filebeat version - 5.6.12

Telnet on port 5044 and 5045 works from the machine where Filebeat is installed to the machine where Logstash is installed

Can someone pls help me with the issue. I have found several documents on the same but did not find he solution for the error.

Thanks!

Eva · May 20, 2019, 7:34am

Filebeat_config

Eva · May 20, 2019, 7:35am

Eva · May 20, 2019, 7:37am

Apologies for using the screenshots for configs as I couldnt set the indentation as I wanted

Eva · May 21, 2019, 5:09am

Can I please get an answer on this request.

Thanks!

Badger · May 21, 2019, 12:33pm

You have logstash configured to use SSL on port 5045, but not 5044. Beats is configured to use SSL and connect to 5044. InvalidFrameProtocolException is expected in that case.

Eva · May 22, 2019, 7:05am

Thanks much, changing the port number worked.

system · June 19, 2019, 7:05am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Getting an error with Logstash/filebeat Logstash	5	461	January 7, 2020
Filebeats cannot push events to logstash Beats filebeat	6	545	October 26, 2021
Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71 Logstash	3	12514	July 24, 2020
Invalid Frame error Logstash	3	843	March 26, 2019
org.logstash.beats.InvalidFrameProtocolException Logstash	2	952	May 26, 2020

InvalidFrameProtocolException and Connection reset by peer error

Errors logged on Logstash :

Related topics