How to reduce filebeat lag time?

Elastic Stack Beats
1

I'm trying to get a near-realtime logstash output from filebeat on windows. The logs are updated frequently, but with periods of no updates for 1-120 seconds at a time. The logging application never produces more than 20 lines/sec.

With my current setup, I am seeing delays of up to 5 seconds between the time the log line is written to when it is shipped to logstash. My project requires this delay to be under 1 second, is this possible?

Here is my config.

filebeat.prospectors:
- input_type: log
  scan_frequency: 1s
  close_inactive: 10m
  ignore_older:   30m
  tail_files: true
  backoff: 0.5s
  max_backoff: 1s
  document_type: hearthstone
  paths:
    - C:/Program Files (x86)/Hearthstone/Logs/Power.log
    - C:/Program Files (x86)/Hearthstone/Logs/Zone.log
output.logstash:
  hosts: ["localhost:5044"]
  workers: 4
2

I would expect your config above to do exactly what you suggest above, just that max could probably be 2s. Perhaps check your logs do see what exactly happens when and where the delay is.

It is possible to even set the above values lower, but you will get a greater CPU overhead as it constantly checks the files.

3

The spooler also has a default flush timeout of 1 second I think.

4

Watching the log on the logstash side:

{:timestamp=>"2017-02-01T11:42:26.939000-0800", :message=>"Beats input: new event received", :event_hash=>{"@metadata"=>{"type"=>"hearthstone", "beat"=>"filebeat"}, "@timestamp"=>"2017-02-01T19:42:22.295Z", "source"=>"C:/Program Files (x86)/Hearthstone/Logs/Zone.log", "offset"=>725589, "type"=>"hearthstone", "input_type"=>"log", "message"=>"D 11:42:21.3677423 ZoneChangeList.ProcessChanges() - TRANSITIONING card [name=River Crocolisk id=23 zone=HAND zonePos=0 cardId=CS2_120 player=1] to FRIENDLY HAND", "beat"=>{"name"=>"WINJAMIE01", "hostname"=>"WINJAMIE01", "version"=>"5.2.0"}}, :identity_stream=>"WINJAMIE01-C:/Program Files (x86)/Hearthstone/Logs/Zone.log", :peer=>"172.16.197.207:49158", :level=>:debug, :file=>"logstash/inputs/beats_support/connection_handler.rb", :line=>"38", :method=>"process"}

Am I reading this correctly?
(Clocks on client/server are in sync)

  • Log line is written at 42:21.3677423
  • Picked up by filebeat at 42:22.295
  • Received by logstash at 42:26.939

So there is a 4 second gap in there. Is it filebeat waiting for more lines before delivering? Or is it logstash slow on the receive?

5

Did you adjust the idle_timeout for the spooler? What does your config look like now?

6

That was what I was missing!
Thanks @ruflin!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.