# GC for persistent queue and LS clustering

**URL:** <https://discuss.elastic.co/t/gc-for-persistent-queue-and-ls-clustering/202400>\
**Category:** Logstash\
**Created:** [October 4, 2019, 11:42pm UTC](https://discuss.elastic.co/t/gc-for-persistent-queue-and-ls-clustering/202400 "2019-10-04T23:42:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![stefsamm](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@stefsamm](https://discuss.elastic.co/u/stefsamm)\
**Post date:** [October 4, 2019, 11:42pm UTC](https://discuss.elastic.co/t/gc-for-persistent-queue-and-ls-clustering/202400/1 "2019-10-04T23:42:17Z")

</div>

I'm workin on switching our log aggregation from Graylog to full ELK setup. While reading on LS clustering found recommendation of adding `queue.checkpoint.writes: 1` to the config to increase durability: [https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html](https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html)

Unfortunately with this setting on, performance drop is unacceptable. It's dropping from average of 15-4K events per second to 500 with gaps between sending events to ES:

Without

 ![14%20PM](https://us1.discourse-cdn.com/elastic/original/3X/2/0/201c7c67b26e61c79155ec58b28ad95f9430e720.png)

With

 ![53%20PM](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3eab20d542985bce4afb516710cb533af710ec89.png)

While queue size is pretty minimal:

 ![50%20PM](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9e9186e4995b064f0d7661848affd6d52e275f81.png)

My guess the culprit is I/O since we're running LS in the cloud with regular EBS backed up root volume.

So first question was: **How important it is to have persistent queues with `queue.checkpoint.writes` set to 1 in case when we want to have multiple instances for each pipeline?**

The other question that I'm struggling to find answer for is GC during persistent queues activated. The same workers/batch.size settings have very different GC performance without (left side) and with (right side) persistent queue:

 ![30%20PM](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ead69a8186e391d25dc73a3b9bbe07e5213d6ad3.png)  
**What is causing such spikes in GC with persistent queues? Is it something that needs to be addressed with other settings/tuning or is expected in this case?**

Will appreciate any feedback. Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 5, 2019, 12:23am UTC](https://discuss.elastic.co/t/gc-for-persistent-queue-and-ls-clustering/202400/2 "2019-10-05T00:23:06Z")

</div>

> [@stefsamm](#):
>
> What is causing such spikes in GC with persistent queues?

It doesn't look like you are measuring spikes in GC, you are measuring spikes in memory usage. Those are a really good thing! My guess is that on the left you are looking at more and more and more unpersisted events being stored on the heap, whilst on the right you are seeing them being stuffed in a queue so that logstash can get on and do something. Rapid sawtooth patterns in memory usage are usually really healthy.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2019, 12:23am UTC](https://discuss.elastic.co/t/gc-for-persistent-queue-and-ls-clustering/202400/3 "2019-11-02T00:23:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.