# How to reduce filebeat lag time?

**URL:** <https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 31, 2017, 9:54pm UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429 "2017-01-31T21:54:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![pixelrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pixelrebel/32/14056_2.png) [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\
**Post date:** [January 31, 2017, 9:54pm UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/1 "2017-01-31T21:54:37Z")

</div>

I'm trying to get a near-realtime logstash output from filebeat on windows. The logs are updated frequently, but with periods of no updates for 1-120 seconds at a time. The logging application never produces more than 20 lines/sec.

With my current setup, I am seeing delays of up to 5 seconds between the time the log line is written to when it is shipped to logstash. My project requires this delay to be under 1 second, is this possible?

Here is my config.

```
filebeat.prospectors:
- input_type: log
  scan_frequency: 1s
  close_inactive: 10m
  ignore_older: 30m
  tail_files: true
  backoff: 0.5s
  max_backoff: 1s
  document_type: hearthstone
  paths:
    - C:/Program Files (x86)/Hearthstone/Logs/Power.log
    - C:/Program Files (x86)/Hearthstone/Logs/Zone.log
output.logstash:
  hosts: ["localhost:5044"]
  workers: 4
```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 1, 2017, 9:40am UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/2 "2017-02-01T09:40:47Z")

</div>

I would expect your config above to do exactly what you suggest above, just that max could probably be 2s. Perhaps check your logs do see what exactly happens when and where the delay is.

It is possible to even set the above values lower, but you will get a greater CPU overhead as it constantly checks the files.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 1, 2017, 1:19pm UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/3 "2017-02-01T13:19:25Z")

</div>

The spooler also has a default flush timeout of 1 second I think.

---

<div class="post-metadata">

**Author:** ![pixelrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pixelrebel/32/14056_2.png) [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\
**Post date:** [February 1, 2017, 7:51pm UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/4 "2017-02-01T19:51:36Z")

</div>

Watching the log on the logstash side:

```
{:timestamp=>"2017-02-01T11:42:26.939000-0800", :message=>"Beats input: new event received", :event_hash=>{"@metadata"=>{"type"=>"hearthstone", "beat"=>"filebeat"}, "@timestamp"=>"2017-02-01T19:42:22.295Z", "source"=>"C:/Program Files (x86)/Hearthstone/Logs/Zone.log", "offset"=>725589, "type"=>"hearthstone", "input_type"=>"log", "message"=>"D 11:42:21.3677423 ZoneChangeList.ProcessChanges() - TRANSITIONING card [name=River Crocolisk id=23 zone=HAND zonePos=0 cardId=CS2_120 player=1] to FRIENDLY HAND", "beat"=>{"name"=>"WINJAMIE01", "hostname"=>"WINJAMIE01", "version"=>"5.2.0"}}, :identity_stream=>"WINJAMIE01-C:/Program Files (x86)/Hearthstone/Logs/Zone.log", :peer=>"172.16.197.207:49158", :level=>:debug, :file=>"logstash/inputs/beats_support/connection_handler.rb", :line=>"38", :method=>"process"}

```

Am I reading this correctly?  
(Clocks on client/server are in sync)

- Log line is written at 42:21.3677423
- Picked up by filebeat at 42:22.295
- Received by logstash at 42:26.939

So there is a 4 second gap in there. Is it filebeat waiting for more lines before delivering? Or is it logstash slow on the receive?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 2, 2017, 8:11am UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/5 "2017-02-02T08:11:09Z")

</div>

Did you adjust the idle\_timeout for the spooler? What does your config look like now?

---

<div class="post-metadata">

**Author:** ![pixelrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pixelrebel/32/14056_2.png) [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\
**Post date:** [February 2, 2017, 8:43am UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/6 "2017-02-02T08:43:44Z")

</div>

> [@ruflin](#):
>
> idle\_timeout

That was what I was missing!  
Thanks @ruflin!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2017, 8:44am UTC](https://discuss.elastic.co/t/how-to-reduce-filebeat-lag-time/73429/7 "2017-03-02T08:44:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.