# FileBeat slow - Improve performance

**URL:** <https://discuss.elastic.co/t/filebeat-slow-improve-performance/144717>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 16, 2018, 12:41pm UTC](https://discuss.elastic.co/t/filebeat-slow-improve-performance/144717 "2018-08-16T12:41:43Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 17, 2018, 1:05pm UTC](https://discuss.elastic.co/t/filebeat-slow-improve-performance/144717/3 "2018-08-17T13:05:29Z")

</div>

I don't see how harvester limit should affect overall ingestion rates. Harvester limit controls how many files you process in parallel.

When trying to tune ingestion, try to identify the bottleneck first. There are a few components that might create back-pressure. Filebeat is rarely the bottleneck.

I'd start testing how fast filebeat can actually process files on localhost. Have a separate filebeat test config, with separate registry file. Delete the registry between runs. Using `pv`, we can check filebeat throughput when using the console output:

```auto
$ ./filebeat -c test-config.yml | pv -Warl >/dev/null

```

in `test-config.yml` we configure the console output:

```auto
output.console:
  pretty: false

```

The `pv`-tool will print the throughput in number of lines == events per second to stderr.

Next try with beats + logstash.

simple logstash config for testing:

```auto
input {
  beat { ... }
}

output {
  stdout { codec => dots }
}

```

This config prints one dot per event. Using pv we can measure throughput:

```auto
./bin/logstash test.cfg | pv -War > /dev/null

```

You can measure the impact of filters in LS, by adding the filters to your `test.cfg`.

If event output rate in LS is large enough, you have to continue tuning LS-\>ES (or just ES).

In filebeat the default queue size is 4096 events and the bulk\_max\_size is 2048. Due to async publishing, you will have at most 4096 events in flight. If sending from one beat only, I don't think `pipeline.batch.size: 8192` will be effective.

In filebeat there is a memory queue (See: [queue docs](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html#configuring-internal-queue)), which collects the events from all harvesters. The queue is used to combine events into batches. The output draws a batch of `bulk_max_size` from the queue. The memory queue size setting is `queue.mem.events`. That is, in filebeat with filled up queues (which is quite normal due to back-pressure), you will have `B = queue.mem.events / output.logstash.bulk_max_size` batches. By default 2. The logstash output by default operates asynchronously, with `pipelining: 2`. That is, one worker can have up to 2 life batches. The queue will block and accept new events only after Logstash did ACK a batch. The total number of output workers in filebeat is given by (assuming `loadbalance: true`): `W = output.logstash.worker * len(hosts)`. The total of batches the set of workers can process is given by `A = W *output.logstash.pipelining`. In order to keep network/outputs busy, we want `queue.mem.events > A * output.logstash.bulk_max_size`, with `queue.mem.events` being a multiple of `bulk_max_size`.

Setting worker \> 1 is similar to configuring the same IP N times in the hosts setting. Assuming you 2 Logstash endpoints and `pipelining: 2` (the default), and `worker: 4` we have: `W = 8` and `A = 16` -\> `queue.mem.events > 32768`. E.g. we could double the number, so to guarantee we have a batch prepare the moment we receive an ACK from logstash -\> `queue.mem.events: 65536`.

If we tune the logstash output in filebeat in any ways without seeing reall improvements, the we did try to tune the wrong sub-system. Always measure first to identify bottlenecks.

Regarding Logstash tuning also check out: [https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html#tuning-logstash](https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html#tuning-logstash)

The `pipeline.batch.size` configures the batch size forwarded to one worker. Having 8 workers, a queue size of 8192, but filebeat just publishing 4096 events max won't give you much of an improvement. A batch of 4096 events likely will be forwarded to one worker only (after some milliseconds delay controlled by `pipeline.batch.delay`). Bigger batches and number of workers will be more helpful if you have multiple filebeat instances.

---

_[View the full topic](https://discuss.elastic.co/t/filebeat-slow-improve-performance/144717)._