InvalidFrameProtocolException and Connection reset by peer error

Hi there

I have been trying to use Filebeat to push the logs to logstash and then output it to AWS ES domain. The logs are shown up on AWS Elasticsearch domain when I do not use SSL authentication between Filebeat and Logstash. But when i try to use SSL mutual authentication [as mentioned here : https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step] between Logstash and Filebeat I see below errors:

Errors logged on Logstash :

[2019-05-20T06:56:04,209][INFO ][org.logstash.beats.BeatsHandler] [local: ip:5044, remote: ip:58800] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
[2019-05-20T06:56:04,209][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 10 more
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 8 more

======================================================

Errors seen on Filebeat :

2019-05-19T07:30:29Z ERR Connecting error publishing events (retrying): read tcp ip:55994->ip:5044: read: connection reset by peer
2019-05-19T07:30:30Z ERR Connecting error publishing events (retrying): read tcp ip:55996->ip:5044: read: connection reset by peer
2019-05-19T07:30:32Z ERR Connecting error publishing events (retrying): read tcp ip:55998->ip:5044: read: connection reset by peer
2019-05-19T07:30:37Z ERR Connecting error publishing events (retrying): read tcp ip:56000->ip:5044: read: connection reset by peer
2019-05-19T07:30:45Z ERR Connecting error publishing events (retrying): read tcp ip:56002->ip:5044: read: connection reset by peer

2019-05-20T06:57:40Z INFO No non-zero metrics in the last 30s
2019-05-20T06:58:05Z ERR Connecting error publishing events (retrying): read tcp ip:58804->ip:5044: read: connection reset by peer
2019-05-20T06:58:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=155
2019-05-20T06:58:40Z INFO No non-zero metrics in the last 30s
2019-05-20T06:59:06Z ERR Connecting error publishing events (retrying): read tcp ip:58806->ip:5044: read: connection reset by peer
2019-05-20T06:59:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=155
2019-05-20T06:59:40Z INFO No non-zero metrics in the last 30s

Here is how my Filebeat config look like:
filebeat.prospectors:

  • paths:
    • /var/log/*.log
      fields:
      application: GS
      account: 1y
      env: prodtest
      name: "GSprod"

output.logstash:
template.name: "filebeat"
template.path: "filebeat.template.json"
hosts: ["www.mydomain.com:5044"]
ssl.certificate_authorities: ["/ssl/ca.crt"]
ssl.certificate: "/sslbeat.crt"
ssl.key: "/beat.key"

Here is how my Logstash config is(removed the filter section as its huge):

input {
beats {
port => 5044
}
beats {
port => 5045
client_inactivity_timeout => 120
ssl => true
ssl_certificate_authorities => ["/opt/ssl/ca.crt"]
ssl_certificate => "/opt/ssl/logstash.crt"
ssl_key => "/opt/ssl/logstash.key"
ssl_verify_mode => "force_peer"
}

}

output {
amazon_es {
hosts => ["vpc-------------es.amazonaws.com"]
region => "Region"
aws_access_key_id => "AK"
aws_secret_access_key => "SK"
index => "logstash-%{+YYYY.MM.dd}"
}
}

Logstash version - logstash 6.7.2
Filebeat version - 5.6.12

Telnet on port 5044 and 5045 works from the machine where Filebeat is installed to the machine where Logstash is installed

Can someone pls help me with the issue. I have found several documents on the same but did not find he solution for the error.

Thanks!

Filebeat_config

image

Apologies for using the screenshots for configs as I couldnt set the indentation as I wanted

Can I please get an answer on this request.

Thanks!

You have logstash configured to use SSL on port 5045, but not 5044. Beats is configured to use SSL and connect to 5044. InvalidFrameProtocolException is expected in that case.

Thanks much, changing the port number worked.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.