# Filebeat can't keep up with logs volume

**URL:** <https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 22, 2021, 7:53pm UTC](https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459 "2021-10-22T19:53:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nsouth](https://avatars.discourse-cdn.com/v4/letter/n/ecccb3/32.png) [@nsouth](https://discuss.elastic.co/u/nsouth)\
**Post date:** [October 22, 2021, 7:53pm UTC](https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459/1 "2021-10-22T19:53:39Z")

</div>

We are trying to run Filebeat on a production server to ship logs to Logstash. However, it is failing to ship logs as quickly as they are being written. If I inspect the Filebeat's registry, it shows that over the past 3 hours it has only processed 26 GB of an 81 GB log file. Our expected volume is 400 - 800 GB of logs per day from this one log file. We are also seeing Filebeat take 60-70% CPU utilization on this server, which is a very beefy server. This makes us wonder if Filebeat can handle the task at hand.

We have checked our Logstash ingester (which sits in front of RabbitMQ) and it does not seem to be bottlenecking. The rest of our pipeline seems healthy.

I thank you for any suggestions.

```auto
filebeat.inputs:
- type: log
  paths:
     - /mypath/log.csv
processors:
- add_fields:
    fields:
      myfield1: "1"
      myfield2: "2"
      myfield3: "3"
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
output.logstash:
  hosts: ["my-load-balancer.domain.com:5045"]
  ssl.certificate_authorities: ["/etc/pki/tls/certs/ca-bundle.crt"]
  worker: 2
  bulk_max_size: 3200

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 22, 2021, 8:39pm UTC](https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459/2 "2021-10-22T20:39:00Z")

</div>

What is your Logstash config pipeline?

The bottleneck could be not on Filebeat, but on Logstash or on the outputs, for example, if you are using elasticsearc as an output and it can't keep up with the events per second rate, it will put back pressure on logstash that will then put back pressure on filebeat.

On most of the time the bottleneck is the logstash outputs or pipeline, not on filebeat.

What are your outputs?

---

<div class="post-metadata">

**Author:** ![nsouth](https://avatars.discourse-cdn.com/v4/letter/n/ecccb3/32.png) [@nsouth](https://discuss.elastic.co/u/nsouth)\
**Post date:** [October 22, 2021, 8:43pm UTC](https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459/3 "2021-10-22T20:43:14Z")

</div>

Filebeat outputs to an ingestion Logstash layer which does not filter and only forwards to RabbitMQ, from which another Logstash layer applies filters and outputs to Elasticsearch. We have not seen any signs of bottlenecks in any of these downstream layers.

---

<div class="post-metadata">

**Author:** ![nsouth](https://avatars.discourse-cdn.com/v4/letter/n/ecccb3/32.png) [@nsouth](https://discuss.elastic.co/u/nsouth)\
**Post date:** [October 24, 2021, 5:45pm UTC](https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459/4 "2021-10-24T17:45:49Z")

</div>

Update: We are in a good state now. I was able to use `exclude_lines` to filter out a lot of noise which didn't need to be shipped and that allowed Filebeat to keep pace. We also were able to decrease the log size by stopping some superfluous processes.

Also, I misspoke when I said that Filebeat took up 60-70% of the CPU. It took up that much on _one core_ of many cores.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2021, 7:46pm UTC](https://discuss.elastic.co/t/filebeat-cant-keep-up-with-logs-volume/287459/5 "2021-11-21T19:46:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.