# Performance hit when multiple filebeats are sending to same ES

**URL:** <https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 8, 2023, 3:30am UTC](https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493 "2023-06-08T03:30:08Z")\
**Posts on this page:** 4\
**Page:** 2

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [June 19, 2023, 11:37am UTC](https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493/22 "2023-06-19T11:37:53Z")

</div>

Hmm, this is even weirder, I would've expected to see higher throughput rates at this point.

A few more ideas:

1. What happens if you disable replicas on the index?
  - Note: This isn't something you should do for production, but I'm curious if it alters your indexing rate.

2. In your `filebeat.yml` when happens if you use the following settings:

```auto
queue.mem.events: 65536
queue.mem.flush.min_events: 2048

output.elasticsearch.bulk_max_size: 500
output.elasticsearch.worker: 16

```

Note: You might want to mess around with the `output.elasticsearch.bulk_max_size` and `output.elasticsearch.worker` settings. Every cluster is different when it comes to the ideal values for [`bulk_max_size`](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_use_bulk_requests) and [`worker`s](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#multiple-workers-threads).

1. What happens now if you have 2 or 3 Filebeats pointed at your 3 node cluster?

---

<div class="post-metadata">

**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\
**Post date:** [June 19, 2023, 3:32pm UTC](https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493/23 "2023-06-19T15:32:48Z")

</div>

> [@](#):
>
> What happens if you disable replicas on the index?

I don't have replicas for my filebeat indices. Only the automatically created indices, e.g. `.ds-.monitoring-es-*`, have replicas.

> [@](#):
>
> In your `filebeat.yml` when happens if you use the following settings

I currently have the following settings.

```auto
queue.mem.events: 64000
queue.mem.flush.min_events: 4000

output.elasticsearch.bulk_max_size: 4000
output.elasticsearch.worker: 8

```

I've tried 12 and 16 workers, but didn't see any difference. I'll play around with the settings again.

From the filebeat logs, it looks like the queue gets filled after about 2 minutes, stays full for about 2-3 minutes, which is when the packet drop happens, then completely empty (gc maybe?), before getting filled up again.

---

<div class="post-metadata">

**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\
**Post date:** [June 27, 2023, 1:09am UTC](https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493/24 "2023-06-27T01:09:44Z")

</div>

I've been trying different combinations of `queue.mem.events`, `queue.mem.flush.min_events` and `output.elasticsearch.bulk_max_size`, but have not seen any significant improvement yet.

When going through the elasticsearch logs, I did see a few `JvmGcMonitorService` warnings on one of the three nodes (the one last added to the cluster). It only appeared about 5 times in the last week, and only on this one node. No searching was done on the ES at the time, only indexing. Not sure if this is relevant.

```auto
[INFO] [o.e.m.j.JvmGcMonitorService] [node-3] [gc] [young] [214426] [26447] duration [854ms], collections [1]/[1.5s], total [854ms]/[7.6m], memory [19.6gb]->[3.1gb]/[30gb], all_pools {[young] [16.5gb]->[16mb]/[0b]}{[old] [3gb]->[3gb]/[30gb]}{[survivor] [81mb]->[64.2mb]/[0b]}
[WARN] [o.e.m.j.JvmGcMonitorService] [node-3] [gc] [young] [214426] overhead, spent [854ms] collecting in the last [1.5s]

```

Each node has 30GB of JVM memory. I currently have a total of 2750 shards, but indexing 8 shards at a time (40GB index, max primary shard size set at 5GB).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2023, 3:10am UTC](https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493/25 "2023-07-25T03:10:18Z")

</div>

Closed on July 25, 2023, 3:10am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/performance-hit-when-multiple-filebeats-are-sending-to-same-es/335493.md?page=1)