# Make Filebeat Faster

**URL:** <https://discuss.elastic.co/t/make-filebeat-faster/140962>\
**Category:** Kibana\
**Created:** [July 20, 2018, 9:02pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962 "2018-07-20T21:02:33Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Naruhodou](https://avatars.discourse-cdn.com/v4/letter/n/7feea3/32.png) [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\
**Post date:** [July 20, 2018, 9:02pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/1 "2018-07-20T21:02:34Z")

</div>

Hi, is there a way to make Filebeat faster? Right now when there are many logs generating at the same time, it takes several seconds before Filebeat pushes to Logstash. Are there any section in the documentation or some configs that can make Filebeat trade more resources for the speed?

Thanks!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 21, 2018, 5:14pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/2 "2018-07-21T17:14:34Z")

</div>

How are you determining this delay? How have you established that Filebeat is the bottleneck? Where is Logstash in turn sending the data?

---

<div class="post-metadata">

**Author:** ![Naruhodou](https://avatars.discourse-cdn.com/v4/letter/n/7feea3/32.png) [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\
**Post date:** [July 21, 2018, 6:58pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/3 "2018-07-21T18:58:18Z")

</div>

Hi Christian,

Thanks for help! I'm not sure whether my way to verify where the delay is is right or not.

Right now Filebeat are running on 10 machines. Logstash is on another machine in the same network. Let's take one line of log as an example.

 ![51%20AM](https://us1.discourse-cdn.com/elastic/original/3X/2/5/2543d0fbb9127799b452f0ad8730d0d11ea964c3.png)

Here the field '@datetime'(11:51:41,387) is when the log is generated. It is parsed from the log. The field '@timestamp'(11:51:44.801) is generated by Filebeat(I think it is?), so I think it's the time when Filebeat sent the message. And the field '@collected\_time'(11:51:45,905) is a field I put at the end of Logstash. The configuration is

```auto
   ruby {
        code => 'require "date"
                 current_time = DateTime.now
                 t = current_time.strftime("%d/%m/%Y %H:%M:%S,%L")
                 event.set("@collected_time", t)'
    }

```

In this case, I think the gap between @timestamp and @datetime is relatively large.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 21, 2018, 10:13pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/4 "2018-07-21T22:13:22Z")

</div>

That seems to be a delay of just a few seconds. This may be because Filebeat batches up events for improved efficiency. How many events are generated per beat per second?

---

<div class="post-metadata">

**Author:** ![Naruhodou](https://avatars.discourse-cdn.com/v4/letter/n/7feea3/32.png) [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\
**Post date:** [July 21, 2018, 10:22pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/5 "2018-07-21T22:22:14Z")

</div>

In the past 5 mins, the average rate is less than 1.(180 logs on 5 machines in 5 mins). But as I see, the speed of generating log is not the same all the time. It's like about 4 logs at the same time, then nothing for several seconds, then another 4 logs at the same time. So the peak rate is about 4 as I see.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 21, 2018, 10:35pm UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/6 "2018-07-21T22:35:53Z")

</div>

I suspect that delay is because you have a very low event rate and Filebeat is buffering. You could lower the [bulk\_max\_size](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_literal_bulk_max_size_literal), but that could limit throughput if volumes were to pick up, as indexing very small batches is inefficient.

---

<div class="post-metadata">

**Author:** ![Naruhodou](https://avatars.discourse-cdn.com/v4/letter/n/7feea3/32.png) [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\
**Post date:** [July 22, 2018, 1:14am UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/7 "2018-07-22T01:14:13Z")

</div>

Which means Filebeat is buffering the events until it reach a certain size to achieve a better performance? Understood. Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2018, 1:14am UTC](https://discuss.elastic.co/t/make-filebeat-faster/140962/8 "2018-08-19T01:14:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.