I've installed filebeat in our k8s following official elastic document (kubernetes/filebeat-kubernetes.yaml )
to collect logs of our microservices and push it to the Logstash which is installed in a different VM as a container, ELK components are installed as separate containers.
filebeat version - 7.17
ELK stack - 7.10
K8s version - v1.23.8
Filebeat starts up perfectly and I see Harvester related information in the logs as well, but on the Logstash side, I see the below error.
Where am I going wrong ? Appreciate any guidance, thanks.
ConfigMap of filebeat
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: logging
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
# hints.default_config:
# type: container
# paths:
# - /var/log/containers/*${data.kubernetes.container.id}.log`
processors:
- add_cloud_metadata:
- add_host_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.logstash:
hosts: ['ELK.HOST.NAME:9192']```
Logstash pipeline's conf
input {
beats {
port => 9192
ssl => false
}
}
filter {
if [kubernetes][container_name] in ["mongo", "kafka"] {
json {
source => "log"
}
if [time] {
date {
match => [ "time" , "UNIX", "ISO8601"]
target => "time"
}
}
}
if "jta" in [kubernetes][container_name] {
json {
source => "log"
target => "api"
remove_field => "log"
}
}
}
output {
if "JIRA" in [kubernetes][container_name] {
elasticsearch {
id => "jira-issue-tracker"
hosts => ["<%= @ipaddress%>:9200"]
index => "jira-issue-tracker-%{[kubernetes][container_name]}-%{+YYYY.MM.dd}"
}
}
}
Logstash logs
[2023-06-05T18:09:57,860][INFO ][org.logstash.beats.BeatsHandler] [local: 172.1.0.2:9192, remote: 10.20.12.162:38784] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
[2023-06-05T18:09:57,863][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-6.0.6.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.11.jar:?]
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.11.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[logstash-input-tcp-6.0.6.jar:?]
... 9 more
[2023-06-05T18:09:57,939][INFO ][org.logstash.beats.BeatsHandler] [local: 172.1.0.2:9192, remote: 10.20.12.162:38784] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
[2023-06-05T18:09:57,940][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-6.0.6.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.11.jar:?]
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.11.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[logstash-input-tcp-6.0.6.jar:?]
... 11 more
[2023-06-05T18:10:32,612][INFO ][org.logstash.beats.BeatsHandler] [local: 172.1.0.2:9192, remote: 10.20.12.162:34442] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
[2023-06-05T18:10:32,612][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [logstash-input-tcp-6.0.6.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-6.0.6.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.11.jar:?]
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.11.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[logstash-input-tcp-6.0.6.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[logstash-input-tcp-6.0.6.jar:?]
... 9 more
[2023-06-05T18:10:32,614][INFO ][org.logstash.beats.BeatsHandler] [local: 172.1.0.2:9192, remote: 10.27.12.162:34442] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
[2023-06-05T18:10:32,614][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69```