Question about filebeat 5.0.0-alpha trial

Elastic Stack Beats
1

I'm using filebeat version 5.0.0-alpha3 (amd64), libbeat 5.0.0-alpha3 for realtime log collection trial in following config:

filebeat.prospectors:

  • input_type: log
    paths:
    • /path/to/access.log.*

output.logstash:
hosts: ["10.0.0.1:24224", "10.0.0.2:24224"]
loadbalance: true

Log record flow is forwarder(filebeat x 3) -> aggregator(td-agent x 2) -> serializer(td-agent x 2) -> file output.

Two questions are as follows:

  1. Most records arrive immediately, but very few record takes 1-2 hours. Can I shorten time by filebeat.yml setting?
  2. When does 5.0.0 become production release?
2

With the td-agent are you referring to the fluentd agent? I'm not aware of the implementation they use so I would recommend to test if everything works as expected with Logstash itself first as this is what we test. I would recommend to update to alpha4 to make sure all the most recent fixes are included.

By default the scan_frequency is 10s in filebeat, so new records should be crawled latest after 10s. I would recommend you to have a look at the filebeat log file to see if there are some output issues in the logs which could explain the delays.