# Logstash - unexpected behaviour with persistent queues

**URL:** <https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395>\
**Category:** Logstash\
**Created:** [January 16, 2019, 1:06am UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395 "2019-01-16T01:06:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sebastian\_Treu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_treu/32/27013_2.png) [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\
**Post date:** [January 16, 2019, 1:06am UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395/1 "2019-01-16T01:06:09Z")

</div>

Hi, I'm testing persisted queues usage and I found that, after a lot of debugging, logstash is being stopped with exit code 0. Logs shows that pipeline is being stopped. No error log found.

I can't find the reason but for sure I found that only is happening when using persisted queue. I've tried with this config:

```auto
      - "queue.type=persisted"
      # I've tried default, 8GB and 10GB without luck
      - "queue.max_bytes=10gb"
      # I've tried default without luck.
      - "queue.page_capacity=256mb"
      - "queue.drain=true"
      - "queue.max_events=0"

```

after realizing that using persisted queues was making logstash stop, I decided to watch the queue size, and this is what I found:

```auto
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
217M	data/queue/
221M	data/queue/
225M	data/queue/
root@localhost:/opt/elk#
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
216M	/usr/share/logstash/data/queue/
220M	/usr/share/logstash/data/queue/
225M	/usr/share/logstash/data/queue/
root@localhost:/opt/elk#
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
220M	/usr/share/logstash/data/queue/
224M	/usr/share/logstash/data/queue/
228M	/usr/share/logstash/data/queue/

```

surprisingly, docker container is being stopped in all cases when queue is almost the same size _- well, no really, logstash is shutting down the pipeline_. I've also monitored the events count and they are around 58K and 61k events.

I'm using this configuration:

```auto
    elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "index"
        query => '{ "query": { "query_string": { "query": "*" } } }'
        scroll => "5m"
        docinfo => true
    }
...
output {
  file {
    path => "/tmp/data.json"
  }
}

```

Using `queue.type=memory` works OK and fetches 310K documents. I'm worried about this behavior has I've set another logstash instances in production to use persisted queues. Any idea on this issue? What am I missing? Any clue/help is appreciated.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Treu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_treu/32/27013_2.png) [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\
**Post date:** [January 16, 2019, 5:23pm UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395/2 "2019-01-16T17:23:30Z")

</div>

I found this topic also in the forum being unresolved: [Logstash stops processing when using persistent queues](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712)

Seems no one has a clue about this issue ☹

---

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [January 16, 2019, 8:39pm UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395/3 "2019-01-16T20:39:05Z")

</div>

That's was me, and I also have a [github issue](https://github.com/elastic/logstash/issues/9722) for the same thing that's being ignored, and is a slightly different problem from the one you're seeing.

In my case, Logstash does not exit, but it refuses to processs any further messages. This is more of a problem than exiting, IMO, as I can't even rely on monitoring the process and externally everything seems to be fine.

Feel free to +1 my issue though 🙂 although I don't hold much hope of anyone doing anything about it after this long.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Treu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_treu/32/27013_2.png) [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\
**Post date:** [January 17, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395/4 "2019-01-17T13:36:59Z")

</div>

@ceekay I see... Not sure why nobody even replies yours or mine posts. Maybe we should provide more info? Is someone using persistent queues in production? I have a bunch of instances with persistent queues and none of them are stalling, however, they are not receiving the same quantity of events as this one.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2019, 1:37pm UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395/5 "2019-02-14T13:37:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.