Filebeat slower than logstash default "file" input?

Elastic Stack Beats
1

Hi,

So far I have been using logstash's default "file" input plugin. I am giving filebeat a try with logstash.

Filebeat gives me a significant performance boost when it comes to reading large logfiles, this is great.

However, I noticed something else:
When appending a single log line manually to logstash through the filebeat input plugin and a simple stdout output without any additional processing in the middle, my payload is taking a lot longer to be processed (I'm talking >5seconds delay complared to logstash input plugin).

My filebeat config is bare minimum. Is there anything I can do to improve this ?

thanks

2

maybe sharing your configuration files and more setup details (OS, filebeat/logsthas version, filesystem) might be a good start.

The 5 second delay sounds like you've enabled multiline support.

3

System:

  • Linux Mint 18
  • filebeat 1.3.1 (amd64)
  • Logstash 2.4.0
  • partition where logs are written is ext4 on a SSD drive

Config files:

$ cat filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - "/tmp/TEST.*.messages.log"

output:
  logstash:
    hosts: ["127.0.0.1:5044"]

$ cat logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
    stdout { codec => rubydebug }
}
```

hope that helps
4

If a file is not updated very frequently, filebeat backs off to not constantly poll the file. I assume before you added the line manually, the file wasn't updated for some time before. You can configure how much filebeat backs off with the config options here: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#_backoff

5

gotcha. good to know, thanks a lot !

6

This topic was automatically closed after 21 days. New replies are no longer allowed.