Filebeat slower than logstash default "file" input?

[remaselvi (Rema)]

Hi,

So far I have been using logstash's default "file" input plugin. I am giving filebeat a try with logstash.

Filebeat gives me a significant performance boost when it comes to reading large logfiles, this is great.

However, I noticed something else:
When appending a single log line manually to logstash through the filebeat input plugin and a simple stdout output without any additional processing in the middle, my payload is taking a lot longer to be processed (I'm talking >5seconds delay complared to logstash input plugin).

My filebeat config is bare minimum. Is there anything I can do to improve this ?

thanks

[steffens (Steffen Siering)]

maybe sharing your configuration files and more setup details (OS, filebeat/logsthas version, filesystem) might be a good start.

The 5 second delay sounds like you've enabled multiline support.

[remaselvi (Rema)]

System:

• Linux Mint 18
• filebeat 1.3.1 (amd64)
• Logstash 2.4.0
• partition where logs are written is ext4 on a SSD drive

Config files:

$ cat filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - "/tmp/TEST.*.messages.log"

output:
  logstash:
    hosts: ["127.0.0.1:5044"]

$ cat logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
    stdout { codec => rubydebug }
}
```

hope that helps

[ruflin]

If a file is not updated very frequently, filebeat backs off to not constantly poll the file. I assume before you added the line manually, the file wasn't updated for some time before. You can configure how much filebeat backs off with the config options here: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#_backoff

[remaselvi (Rema)]

gotcha. good to know, thanks a lot !