# Logstash stops processing when using persistent queues

**URL:** <https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712>\
**Category:** Logstash\
**Created:** [June 6, 2018, 1:30am UTC](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712 "2018-06-06T01:30:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [June 6, 2018, 1:30am UTC](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712/1 "2018-06-06T01:30:39Z")

</div>

I've updated logstash.yml to use persistent queues for a bit of extra redundancy, and have run into a problem where individual nodes will just stop processing messages after ~~about 30+ minutes~~ a random amount of time.

Kafka also shows the partition they're reading from as stopped.

The nodes themselves are still doing something, as they are still producing metrics, so I suspect it's related to the Kafka input somehow.

Once a node stops, the remaining nodes will pick up the messages from its Kafka partition after a ~5 minute delay, however if left long enough, all the nodes will eventually stop processing.

I really have no idea where to start looking as there is nothing of note in logstash-plain.log.

logstash.yml changes are:

```auto
queue.type: persisted
queue.page_capacity: 10mb
queue.max_bytes: 10mb
queue.checkpoint.writes: 256

```

If I disable persistent queues, Logstash goes back to behaving itself. Enable queues again, and I get more stoppages.

Any ideas what could be causing this?

Logstash 6.2.2-1  
Kafka 2.11-0.11.0.2

---

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [June 7, 2018, 10:47pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712/2 "2018-06-07T22:47:20Z")

</div>

Bump: any ideas? This is unusable for me at the moment.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2018, 10:47pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712/3 "2018-07-05T22:47:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.