Hi there
I have been trying to use Filebeat to push the logs to logstash and then output it to AWS ES domain. The logs are shown up on AWS Elasticsearch domain when I do not use SSL authentication between Filebeat and Logstash. But when i try to use SSL mutual authentication [as mentioned here : https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step] between Logstash and Filebeat I see below errors:
Errors logged on Logstash :
[2019-05-20T06:56:04,209][INFO ][org.logstash.beats.BeatsHandler] [local: ip:5044, remote: ip:58800] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
[2019-05-20T06:56:04,209][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 10 more
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 8 more
======================================================
Errors seen on Filebeat :
2019-05-19T07:30:29Z ERR Connecting error publishing events (retrying): read tcp ip:55994->ip:5044: read: connection reset by peer
2019-05-19T07:30:30Z ERR Connecting error publishing events (retrying): read tcp ip:55996->ip:5044: read: connection reset by peer
2019-05-19T07:30:32Z ERR Connecting error publishing events (retrying): read tcp ip:55998->ip:5044: read: connection reset by peer
2019-05-19T07:30:37Z ERR Connecting error publishing events (retrying): read tcp ip:56000->ip:5044: read: connection reset by peer
2019-05-19T07:30:45Z ERR Connecting error publishing events (retrying): read tcp ip:56002->ip:5044: read: connection reset by peer
2019-05-20T06:57:40Z INFO No non-zero metrics in the last 30s
2019-05-20T06:58:05Z ERR Connecting error publishing events (retrying): read tcp ip:58804->ip:5044: read: connection reset by peer
2019-05-20T06:58:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=155
2019-05-20T06:58:40Z INFO No non-zero metrics in the last 30s
2019-05-20T06:59:06Z ERR Connecting error publishing events (retrying): read tcp ip:58806->ip:5044: read: connection reset by peer
2019-05-20T06:59:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=155
2019-05-20T06:59:40Z INFO No non-zero metrics in the last 30s
Here is how my Filebeat config look like:
filebeat.prospectors:
- paths:
- /var/log/*.log
fields:
application: GS
account: 1y
env: prodtest
name: "GSprod"
- /var/log/*.log
output.logstash:
template.name: "filebeat"
template.path: "filebeat.template.json"
hosts: ["www.mydomain.com:5044"]
ssl.certificate_authorities: ["/ssl/ca.crt"]
ssl.certificate: "/sslbeat.crt"
ssl.key: "/beat.key"
Here is how my Logstash config is(removed the filter section as its huge):
input {
beats {
port => 5044
}
beats {
port => 5045
client_inactivity_timeout => 120
ssl => true
ssl_certificate_authorities => ["/opt/ssl/ca.crt"]
ssl_certificate => "/opt/ssl/logstash.crt"
ssl_key => "/opt/ssl/logstash.key"
ssl_verify_mode => "force_peer"
}
}
output {
amazon_es {
hosts => ["vpc-------------es.amazonaws.com"]
region => "Region"
aws_access_key_id => "AK"
aws_secret_access_key => "SK"
index => "logstash-%{+YYYY.MM.dd}"
}
}
Logstash version - logstash 6.7.2
Filebeat version - 5.6.12
Telnet on port 5044 and 5045 works from the machine where Filebeat is installed to the machine where Logstash is installed
Can someone pls help me with the issue. I have found several documents on the same but did not find he solution for the error.
Thanks!