# "Wild swings" in event rate when number of workers increase

**URL:** https://discuss.elastic.co/t/wild-swings-in-event-rate-when-number-of-workers-increase/297320
**Category:** Beats
**Tags:** elastic-stack-monitoring, filebeat
**Created:** [February 16, 2022, 4:38am UTC](https://discuss.elastic.co/t/wild-swings-in-event-rate-when-number-of-workers-increase/297320 "2022-02-16T04:38:38Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)
#### Post date: [February 16, 2022, 4:38am UTC](https://discuss.elastic.co/t/wild-swings-in-event-rate-when-number-of-workers-increase/297320/1 "2022-02-16T04:38:38Z")

</div>

Hi,

I'm trying to increase the event rate for my filebeat that is writing Netflow data to Elasticsearch, to keep up with the incoming Netflow traffic. I have already increased `max_bulk_size` to 1000 in `filebeat.yml`, which seems to have tripled my max event rate from 1600/s to 5000/s. Next, I'm trying to increase the number of workers writing to ES.

I found that when I have `worker: 1`, the event rate stays above 0. But when I increase the number of workers to 2 or 4, while the max event rate increased, the min event rate goes to 0 periodically (min-max difference is much larger than when `worker: 1`). Is this to be expected?

I have created the graphs below to illustrate this (ignore the actual numbers as this is not the actual graphs, I just wanted to show the difference).

 ![picture](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5dac5fb7a504f821abf5613c9a7ef6994807b9e5.png)

BTW, the CPU utilization also stays above 100% (fluctuates between 120-150%). Not sure if this is normal.

Since the increase in `max_bulk_size`, I'm seeing about 150K events every 1 minute, instead of the previous 150K events every 2 minutes with default `max_bulk_size` and `worker` values. Should I expect the interval between events to be even lower, or more events every minute, as I'm definitely getting more events that this?

I'm using a server with 48 cores, and have ringfenced 64GB of RAM for ES (free memory is around 20GB).

Thank you.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 16, 2022, 6:38am UTC](https://discuss.elastic.co/t/wild-swings-in-event-rate-when-number-of-workers-increase/297320/2 "2022-03-16T06:38:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.