Logstash - unexpected behaviour with persistent queues

[Sebastian_Treu]

Hi, I'm testing persisted queues usage and I found that, after a lot of debugging, logstash is being stopped with exit code 0. Logs shows that pipeline is being stopped. No error log found.

I can't find the reason but for sure I found that only is happening when using persisted queue. I've tried with this config:

      - "queue.type=persisted"
      # I've tried default, 8GB and 10GB without luck
      - "queue.max_bytes=10gb"
      # I've tried default without luck.
      - "queue.page_capacity=256mb"
      - "queue.drain=true"
      - "queue.max_events=0"

after realizing that using persisted queues was making logstash stop, I decided to watch the queue size, and this is what I found:

root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
217M	data/queue/
221M	data/queue/
225M	data/queue/
root@localhost:/opt/elk#
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
216M	/usr/share/logstash/data/queue/
220M	/usr/share/logstash/data/queue/
225M	/usr/share/logstash/data/queue/
root@localhost:/opt/elk#
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
220M	/usr/share/logstash/data/queue/
224M	/usr/share/logstash/data/queue/
228M	/usr/share/logstash/data/queue/

surprisingly, docker container is being stopped in all cases when queue is almost the same size - well, no really, logstash is shutting down the pipeline. I've also monitored the events count and they are around 58K and 61k events.

I'm using this configuration:

    elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "index"
        query => '{ "query": { "query_string": { "query": "*" } } }'
        scroll => "5m"
        docinfo => true
    }
...
output {
  file {
    path => "/tmp/data.json"
  }
}

Using queue.type=memory works OK and fetches 310K documents. I'm worried about this behavior has I've set another logstash instances in production to use persisted queues. Any idea on this issue? What am I missing? Any clue/help is appreciated.

[Sebastian_Treu]

I found this topic also in the forum being unresolved: Logstash stops processing when using persistent queues

Seems no one has a clue about this issue

[ceekay]

That's was me, and I also have a github issue for the same thing that's being ignored, and is a slightly different problem from the one you're seeing.

In my case, Logstash does not exit, but it refuses to processs any further messages. This is more of a problem than exiting, IMO, as I can't even rely on monitoring the process and externally everything seems to be fine.

Feel free to +1 my issue though although I don't hold much hope of anyone doing anything about it after this long.

[Sebastian_Treu]

@ceekay I see... Not sure why nobody even replies yours or mine posts. Maybe we should provide more info? Is someone using persistent queues in production? I have a bunch of instances with persistent queues and none of them are stalling, however, they are not receiving the same quantity of events as this one.