Logstash Beats Error [ELK Stack 6.5.4]

Hey folks,

after a long time away from the elk stack I wanted to set up another testing scenario. I followed the latest stack guide from here: https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html.

My PROBLEM:

connecting beats <-> logstash. The error msg is following beneath of the description.

What is working: Metric-/Filebeat to Kibana/Elasticsearch
What doesn't work: Filebeat via Logstash to Elesticsearch/Kibana

Without logstash everything works like a charm. I followed and crosschecked everything twice. I even tried a third party guide from digital ocean: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-16-04 on a clean vm.

In case i missed to provide some informations I'll deliver them as fast as possible.

ELK
All my elk components are in version 6.5.4 and installed from repo

debian:

Linux elk-stack2 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux

java -version

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

filebeat.yml

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

Logstash Pipeline:

input {
  beats {
    port => 5044
  }
}

# The filter part of this file is commented out to indicate that it
# is optional.
# filter {
#
# }

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Logstash Error log:

Jan 20 20:54:12 elk-stack2 logstash[1687]: [2019-01-20T20:54:12,444][INFO ][org.logstash.beats.BeatsHandler] [local: 0:0:0:0:0:0:0:1:5044, remote: 0:0:0:0:0:0:0:1:60972] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
    Jan 20 20:54:12 elk-stack2 logstash[1687]: [2019-01-20T20:54:12,444][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    Jan 20 20:54:12 elk-stack2 logstash[1687]: io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
    Jan 20 20:54:12 elk-stack2 logstash[1687]: Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.6.jar:?]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
    Jan 20 20:54:12 elk-stack2 logstash[1687]:         ... 10 more

only a slight misconfiguration on the filebeat side
comment in the right output and all should be working
(comment out the output.Elasticsearch and comment in output.logstash)

Have a look at this thread

Also in you filbeat.yml you have no # in front of your hosts in the logstash output.

If you output to logstash your config should look like this:

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

and you should disable the Elasticsearch output:

Add a # infront of the output.Elasticsearch

output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

Hope this helps,

Paul.

Thank you so much. I'm kinda ashamed right now. I wasn't able to see the forest for the trees...

Can be closed :slight_smile:

1 Like

No worries, it can happen to the best of us..

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.