Logstash not getting data from fliebeat . Logstash logs are empty

Elastic Stack Beats
1

Hello,
I am trying to send some logs from filebeat to logstash output.
problem is beats is sending the events to the logstash server (I can see in the debug logs
). But log stash does not see anything coming . Even the log stash log files are empty. LS just shows a startup message.
Filebeat version 1.2.3-1
Logstash version 2.3.4

Filebeat.yml

filebeat:
prospectors:
-
paths:
- /var/log/secure
input_type: log
document_type: fuse
multiline:
pattern: ^[
#negate: false
match: after

output:
logstash:
hosts: ["localhost:5200"]
index: secure

logging:
files:
rotateeverybytes: 10485760 # = 10MB

Logstash.conf file

Specify input from filebeat

input {
beats {
type => "beats"
port => 5200
}
}
output {

if "fuse-logs" in [tags] {

    elasticsearch {
        hosts => "localhost:9200"
        template_name => "fuse-logs"
    }
#}

I can see successful messages of sent event in the beats logs .
filebeat -c filebeat.yml -e -d "*"
Filebeat logs --
2016/07/21 03:40:53.740082 output.go:87: DBG output worker: publish 1 events
2016/07/21 03:40:53.740112 client.go:146: DBG Try to publish 1 events to logstash with window size 10
2016/07/21 03:40:53.744620 client.go:124: DBG 1 events out of 1 events sent to logstash. Continue sending ...
2016/07/21 03:40:53.744635 single.go:135: DBG send completed
2016/07/21 03:40:53.744659 publish.go:104: INFO Events sent: 1
2016/07/21 03:40:53.744682 registrar.go:115: DBG Processing 1 events
2016/07/21 03:40:53.744697 registrar.go:146: DBG Write registry file: /etc/filebeat/.filebeat
2016/07/21 03:40:53.744892 registrar.go:162: INFO Registry file updated. 1 states written.
2016/07/21 03:40:56.255302 reader.go:138: DBG End of file reached: /var/log/secure; Backoff now.
2016/07/21 03:41:01.239876 spooler.go:97: DBG Flushing spooler because of timeout. Events flushed: 0
2016/07/21 03:41:01.254924 prospector.go:185: DBG Start next scan
2016/07/21 03:41:01.254941 prospector.go:261: DBG scan path /var/log/secure
2016/07/21 03:41:01.254962 prospector.go:275: DBG Check file for harvesting: /var/log/secure

But Logstash does not get the events from beat. LS is running. Not writing anything in the logstash.log file.
cat /var/log/logstash/logstash.log
{:timestamp=>"2016-07-20T10:23:55.707000-0400", :message=>"Pipeline main started"}
{:timestamp=>"2016-07-20T10:32:19.540000-0400", :message=>"Pipeline main started"}
{:timestamp=>"2016-07-20T23:13:22.929000-0400", :message=>"Pipeline main started"}

Error log is also empty.

Kindly help what could be the problem here.
Thanks in advance

2

please properly format code sections (e.g. using </> button).

if the configured path correct? Like is it a file or a directory? filebeat requires glob-pattern for files.

Have you tried without multiline first?

How big is your file?

Something has been send and ACKed by logstash:

2016/07/21 03:40:53.740082 output.go:87: DBG output worker: publish 1 events
2016/07/21 03:40:53.740112 client.go:146: DBG Try to publish 1 events to logstash with window size 10
2016/07/21 03:40:53.744620 client.go:124: DBG 1 events out of 1 events sent to logstash. Continue sending ...
2016/07/21 03:40:53.744635 single.go:135: DBG send completed
2016/07/21 03:40:53.744659 publish.go:104: INFO Events sent: 1
2016/07/21 03:40:53.744682 registrar.go:115: DBG Processing 1 events

but then no new content has been added to file?

2016/07/21 03:40:56.255302 reader.go:138: DBG End of file reached: /var/log/secure; Backoff now.
3

@steffens

Thank you for replying.
I ran the logstash in debug mode from command line and found that logs had a field <@timestamp/> apart from timestamp. This resulted in error and LS was not sending the data to ES.
We added field <@timestamp/> in the grok filter and solved the problem.
Thank you very much for replying :slight_smile:

4

This topic was automatically closed after 21 days. New replies are no longer allowed.