Minimal logstash pipeline for beats (6.x)

[pas]

Hi,

I am currently in the beginning of learning the ELK-stack, so have me excused for asking for the basics.

My first goal is to set up a functional beats(winlogbeat and filebeat), logstash, kibana and elasticsearch environment without any advanced configs. I also try to go for the "current" version of all the components, that is v6.x .

Until now I've managed to set up everything, but logstash.. which is now giving me a bit of headache. When pushing logs directly from winlogbeat (a Win10 desktop) and filebeat (ubuntu 16.04LTS server logs, as kern.log and auth.log) in to Elasticsearch, it works fine and is presented as expected in Kibana.

Although, my issue now is setting up the correct Logstash-pipelines for these "beats".

Can anybody show me an example of a minimal functional pipeline for this? Or at least let me know what is the minimal required config for respectively input, filter and output?

[yaauie (Ry Biesemeyer)]

• A valid pipeline will need at least one input and one output; filters are optional.
• The only required parameter for the Beats Input is the port; options and their default values are documented here
• The Elasticsearch Output Plugin options and their default values are documented here

The minimal configuration then, would look something like this:

input {
  beats {
    port => 1234
  }
}

filter {
  # filters would go here once you have need for them
}

output {
  elasticsearch {
    hosts => ["http://192.168.1.127:9200","http://192.168.1.128:9200"]
  }
}

[pas]

Thank you so much for replying!

I agree to what you are describing and I've already tried such pipeline config. But it still doesnt work...

I'll posted my config below if anyone could help me sweep through to look for obvious mistakes:) I only paste the uncommented lines (IPs are removed in paste, but are verified correct in config)

My logstash-plain.log gives me this when i start it:

[2018-05-15T09:15:23,785][INFO ][org.logstash.beats.BeatsHandler] [local: removed:5044, remote: removed:34694] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2018-05-15T09:15:23,785][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.0.13.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	... 10 more

I'll show you my config to narrow it down;

elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: removed

logstash.yml

path.data: /var/lib/logstash
path.logs: /var/log/logstash

pipelines.yml

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

logstash-beats.conf (located at /etc/logstash/conf.d/)

input {
  beats {
    port => 5044
  }
}

filter {
  #not for now
}

output {
  elasticsearch { hosts => [" removed :9200"] }
}

filebeat.yml

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "removed:5601"

output.logstash:
  hosts: "removed:5044"

[pas]

Well. I totally fell into the "find your own mistakes while writing a community-forum post"...

The error was due to faulty config in both my winlogbeat and filebeat where I hadnt uncommented the line with "output.logstash:" and commented the "output.elasticsearch"... I had only changed the line with "hosts:" within these sections.

Anyway, big thanks for replying!

[yaauie (Ry Biesemeyer)]

@pas

"find your own mistakes while writing a community-forum post"

I'm glad I could be of help. Sometimes the act of describing a problem to someone else forces us to see the situation in a new light, which can be immensely helpful in and of itself.

That said, it looks like something is attempting to establish a connection with the TCP port that Logstash opened up for your Beats input, and the payload of that connection is not valid Beats protocol. From your pasted output, it appears as though this is causing the input to crash. If you encounter the issue again, you may want to open up an issue on the Beats Input plugin so we can chase it down.

[pas]

You are absolutely right:) Thanks again!