Hi, I'm testing persisted queues usage and I found that, after a lot of debugging, logstash is being stopped with exit code 0. Logs shows that pipeline is being stopped. No error log found.
I can't find the reason but for sure I found that only is happening when using persisted queue. I've tried with this config:
- "queue.type=persisted"
# I've tried default, 8GB and 10GB without luck
- "queue.max_bytes=10gb"
# I've tried default without luck.
- "queue.page_capacity=256mb"
- "queue.drain=true"
- "queue.max_events=0"
after realizing that using persisted queues was making logstash stop, I decided to watch the queue size, and this is what I found:
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
217M data/queue/
221M data/queue/
225M data/queue/
root@localhost:/opt/elk#
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
216M /usr/share/logstash/data/queue/
220M /usr/share/logstash/data/queue/
225M /usr/share/logstash/data/queue/
root@localhost:/opt/elk#
root@localhost:/opt/elk# docker exec -it logstash bash -c "while true; do sleep 2; du -hs /usr/share/logstash/data/queue/; done"
...
220M /usr/share/logstash/data/queue/
224M /usr/share/logstash/data/queue/
228M /usr/share/logstash/data/queue/
surprisingly, docker container is being stopped in all cases when queue is almost the same size - well, no really, logstash is shutting down the pipeline. I've also monitored the events count and they are around 58K and 61k events.
I'm using this configuration:
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "index"
query => '{ "query": { "query_string": { "query": "*" } } }'
scroll => "5m"
docinfo => true
}
...
output {
file {
path => "/tmp/data.json"
}
}
Using queue.type=memory
works OK and fetches 310K documents. I'm worried about this behavior has I've set another logstash instances in production to use persisted queues. Any idea on this issue? What am I missing? Any clue/help is appreciated.