# Increase performance for a PubSub module in Filebeat

**URL:** <https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 4, 2019, 6:22am UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466 "2019-12-04T06:22:34Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![daniel\_a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_a/32/48390_2.png) [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\
**Post date:** [December 4, 2019, 6:22am UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466/1 "2019-12-04T06:22:34Z")

</div>

How to improve performance for the Google Cloud module on a filebeat? I'm using Google Cloud module running on a filebeat: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-googlecloud.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-googlecloud.html), and my event rate is super low.

Any thoughts how to tune it? Setting **bulk\_max\_size** or **worker** doesn't help at all.

I have set up this but it didn't help.

```
output.elasticsearch:
  bulk_max_size: 3200
  worker: 2
```

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [December 5, 2019, 10:40pm UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466/2 "2019-12-05T22:40:54Z")

</div>

Hi @daniel_a, what does your config look like for filebeat?

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [December 5, 2019, 10:41pm UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466/3 "2019-12-05T22:41:27Z")

</div>

Maybe @andrewkroh can help here 🙂

---

<div class="post-metadata">

**Author:** ![daniel\_a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_a/32/48390_2.png) [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\
**Post date:** [December 10, 2019, 6:27pm UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466/4 "2019-12-10T18:27:57Z")

</div>

```
cloud.id: "${CLOUD_ID}"
cloud.auth: "${CLOUD_AUTH}"

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression

#flow__logs_template has 6 shards
setup.template.name: "flow__logs_template"
setup.template.overwrite: true

tags: ["flow_logs"]

#How to Tune Elastic Beats Performance: A Practical Example with Batch Size, Worker Count
 # - https://www.elastic.co/blog/how-to-tune-elastic-beats-performance-a-practical-example-with- 
  # batch-size-worker-count-and-more

#set queue.mem.events to 2 * workers * batch size
queue.mem.events: 115200
#set queue.mem.flush.min_events to batch size
queue.mem.flush.min_events: 2400

output.elasticsearch:
   bulk_max_size: 2400
   worker: 24
   compression_level: 9

indices:
  - index: "vpc_flow_logs"
    when.contains:
      event.type: "flow"
```

---

<div class="post-metadata">

**Author:** ![daniel\_a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_a/32/48390_2.png) [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\
**Post date:** [December 10, 2019, 10:09pm UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466/5 "2019-12-10T22:09:57Z")

</div>

++ @andrewkroh

I can barely get above 6,000 EPS/s with the filebeat and googlecloud module. The best I did was 6,487 EPS/s. Server CPU's utilization was about 25% 😞

I started with the blog post: [https://www.elastic.co/blog/how-to-tune-elastic-beats-performance-a-practical-example-with-batch-size-worker-count-and-more](https://www.elastic.co/blog/how-to-tune-elastic-beats-performance-a-practical-example-with-batch-size-worker-count-and-more). Next, I set up the following configs (after testing many different ones first):

```
#set queue.mem.events to 2 * workers * batch size
queue.mem.events: 115200
#set queue.mem.flush.min_events to batch size
queue.mem.flush.min_events: 2400

output.elasticsearch:
   bulk_max_size: 2400
   worker: 24
   #this is optional, I tested it with 0, 5, and finally with 9
   compression_level: 9

```

I then looked at the googlecloud module (pubsub connection), and I added the following configs:

```
subscription.max_outstanding_messages: 120000
subscription.num_goroutines: 32

```

For the googlecloud module, I also tested different configs (large numbers or even -1), some of that was described in here [https://github.com/googleapis/google-cloud-go/wiki/Fine-Tuning-PubSub-Receive-Performance#subscriptionreceivesettingsmaxoutstandingmessages](https://github.com/googleapis/google-cloud-go/wiki/Fine-Tuning-PubSub-Receive-Performance#subscriptionreceivesettingsmaxoutstandingmessages)).

And, I can't get anything above 6,487 EPS/s, any thoughts?

Is there a way to measure the filebeat performance after deploying each configuration change other than looking into the Kibana metric dashboard?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 7, 2020, 10:11pm UTC](https://discuss.elastic.co/t/increase-performance-for-a-pubsub-module-in-filebeat/210466/6 "2020-01-07T22:11:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.