# How to increase filebeat speed

**URL:** <https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 12, 2019, 1:02pm UTC](https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254 "2019-09-12T13:02:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmacew](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@kmacew](https://discuss.elastic.co/u/kmacew)\
**Post date:** [September 12, 2019, 1:02pm UTC](https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254/1 "2019-09-12T13:02:35Z")

</div>

Dear elastic team,

In my environment i got around 6-7 applications. These applications logs around 30-40 lines per second, it's few GB per day. Filebeat can't keep up with parsing logs to send them to elasticsearch (via Logstash). I tried to increase speed of filebeat by adding additional flags without success.

My filebeat version is:  
filebeat version 7.3.1 (amd64), libbeat 7.3.1 [a4be71b90ce3e3b8213b616adfcd9e455513da45 built 2019-08-19 19:30:50 +0000 UTC]

and my config:

filebeat.spool\_size: 8192  
filebeat.publish\_async: true

filebeat.inputs:

- type: log  
enabled: true  
paths:
  - /opt/\*.log  
include\_lines: ['ERROR']  
close\_removed: true

output.logstash:  
hosts: ["logstash:5048", "logstash:5054", "logstash:5055"]  
bulk\_max\_size: 8192  
loadbalance: true  
worker: 3

Is there any chance to increase speed of filebeat parse?

Regards,  
Krzysztof

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 13, 2019, 5:56am UTC](https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254/2 "2019-09-13T05:56:22Z")

</div>

At that throughput level it sounds unlikely that Filebeat is the bottleneck. Filebeat can only send as fast as Logstash and downstream systems can accept. How have you determined that Filebeat is the bottleneck?

---

<div class="post-metadata">

**Author:** ![kmacew](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@kmacew](https://discuss.elastic.co/u/kmacew)\
**Post date:** [September 13, 2019, 9:28am UTC](https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254/3 "2019-09-13T09:28:55Z")

</div>

Thanks you Christian for reply.

I started filebeat in debug mode with command:

sudo filebeat -c /etc/filebeat/filebeat.yml -d "publish"

In field 'message' i saw that timestamp from my log is much older than real time, with the time the distance between two times were getting bigger. Example log:

11:26:12.590 INFO xxx

Regards,  
Krzysztof

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [September 13, 2019, 7:06pm UTC](https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254/4 "2019-09-13T19:06:05Z")

</div>

What is the scan\_frequency param value ?

Also, to verify that Filebeat is the bottleneck, you can also try to send the output of Filebeat to console

> output.console:  
> enabled: true  
> pretty: true

make sure to disable other output modes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2019, 7:06pm UTC](https://discuss.elastic.co/t/how-to-increase-filebeat-speed/199254/5 "2019-10-11T19:06:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.