Logstash is not capturing data from file beat

Elastic Stack Logstash
1

[2018-04-09T16:57:23,836][INFO ][org.logstash.beats.BeatsHandler] [local: 192.168.36.198:5044, remote: 192.168.36.37:47240] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2018-04-09T16:57:23,836][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.0.10.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]

2

What does your Logstash and Filebeat configurations look like? Make sure you posted them as preformatted text (e.g. using the </> toolbar button).

3

Logstash config File:
input {
beats {
port => 5044
type => "log"
}
}

filter {
grok {
match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}.log" }
}
}

output {
elasticsearch {
hosts => "192.168.36.37:9200"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

Filebeat Config file :

#----------------------------- Logstash output --------------------------------
#output.logstash:

The Logstash hosts

hosts: ["192.168.36.37:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/*
      #- c:\programdata\elasticsearch\logs*
4

and i can see logstash service running but lot listening on port number 5044 .

5

I repeat: Make sure you post them as preformatted text (e.g. using the </> toolbar button).

6 
indent preformatted text by 4 spaces

Logstash config File:
input {
beats {
port => 5044
type => "log"
}
}

filter {
grok {
match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}.log" }
}
}
output {
elasticsearch {
hosts => "192.168.36.37:9200"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

7 
indent preformatted text by 4 spaces

Filebeat Config file :

#----------------------------- Logstash output --------------------------------
#output.logstash:
The Logstash hosts

hosts: ["192.168.36.37:5044"]
Optional SSL. By default is off.
List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"
Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#=========================== Filebeat prospectors =============================

filebeat.prospectors:
Each - is a prospector. Most options can be set at the prospector level, so
you can use different prospectors for various configurations.
Below are the prospector specific configurations.

type: log
Change to true to enable this prospector configuration.

enabled: true
Paths that should be crawled and fetched. Glob based paths.

paths:
    /var/log/*
    #- c:\programdata\elasticsearch\logs
indent preformatted text by 4 spaces
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.