# LS loosing messages with persisted queue **URL:** \ **Category:** Logstash\ **Created:** [July 20, 2017, 5:54am UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868 "2017-07-20T05:54:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Post date:** [July 20, 2017, 5:54am UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868/1 "2017-07-20T05:54:22Z")
Hello Logstash experts, I've encountered a very weird issue: I was indexing a database using the JDBC input, but realized, what the document count in ES was less than in the source table. It tuned out, that the reason was the persisted queue - with the memory queue the document count matched. So, I tried to reproduced the issue with a very basic configuration: `logstash.yml` ```auto queue.type: memory #queue.type: persisted log.level: info path.logs: c:/temp/ls_logs path.data: c:/temp/ls_data ``` 1. with `queue.type: memory` ```auto C:\user\programms\logstash>bin\logstash --path.settings file:///C:/user/etc/logstash_test -e "input { generator { count => 10000 } } filter {uuid {target => '@uuid'} } output {stdout {codec => line}}" -w 1 -b 1 | find /c "Hello" 10000 ``` 1. with `queue.type: persisted` ```auto C:\user\programms\logstash>bin\logstash --path.settings file:///C:/user/etc/logstash_test -e "input { generator { count => 10000 } } filter {uuid {target => '@uuid'} } output {stdout {codec => line}}" -w 1 -b 1 | find /c "Hello" 1759 ``` 1. with `queue.type: persisted` once again ```auto C:\user\programms\logstash>bin\logstash --path.settings file:///C:/user/etc/logstash_test -e "input { generator { count => 10000 } } filter {uuid {target => '@uuid'} } output {stdout {codec => line}}" -w 1 -b 1 | find /c "Hello" 1710 ``` 1. and back to `queue.type: memory` ```auto C:\user\programms\logstash>bin\logstash --path.settings file:///C:/user/etc/logstash_test -e "input { generator { count => 10000 } } filter {uuid {target => '@uuid'} } output {stdout {codec => line}}" -w 1 -b 1 | find /c "Hello" 10000 ``` Logstash version: 5.5.0 OS: Windows 10 Java: 1.8 Does someone have an idea, what could be wrong here? Thanks! ---
**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Post date:** [July 20, 2017, 6:01am UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868/2 "2017-07-20T06:01:30Z")
it must be something wrong with the worker count: `-w 2 -b 100` =\> `9802` lines `-w 2 -b 1000` =\> `11700` lines using the previous command and with `queue.type: persisted` and w/o the worker parameter: `-b 2000 => 10000` `-b 10 => 10000` Update: it seems, that the worker count and the batch size are irrelevant, because sometimes it works with the same settings and sometimes does not. ---
**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Post date:** [July 28, 2017, 10:27am UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868/3 "2017-07-28T10:27:41Z")
Sorry for bumping up the thread, but it seems to be a very important issue. I could reproduce it on Unix as well (redhat). Make sense to open a Github ticket? Thanks! ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [July 31, 2017, 2:03pm UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868/4 "2017-07-31T14:03:07Z")
Your example using generator to illustrate the problem does not illustrate it at all... The generator input is the one and only input that shuts down LS when its done. With memory queue, as its synchronous, when the last 10000th event is produced the filter/output will process it - but with PQ these two functions are disconnected so the input rapidly produces 10000 events and persists them then begins the shutdown, The filter/output stage only processes 1750 ish events before it is shutdown, the rest are in the PQ for next time. This is by design. There will be a `queue.drain` option available soon to allow for ephemeral container based instances to drain the queue after receiving an ordered shutdown command. Back to your original problem. Are you shutting down LS after the JDBC input has read the DB contents (but before the filter/outputs have read all events out of the queue? If you look at the `path.queue` contents, you should see the queue growing while the inputs write faster than the filters read but once the inputs have stopped the contents should shrink until there two files left - `NNN.page` file and its checkpoint file. ---
**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Post date:** [July 31, 2017, 7:11pm UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868/5 "2017-07-31T19:11:59Z")
thank you very much for getting back to me! I see, my "generator" example was not a proper way to reproduce the issue. I cannot run the original tests currently, but as far as I remember, I was shutting down the LS instance by `Ctrl-C`. The size of the `queue` directory was 256Mb - basically it contained one page file, but I need to double check this again. And thanks a lot for the explanation how the PQ works. That explains as well, why sometimes the number or processed events were more 10.000. I'll re-test with the JDBC input and let you know. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2017, 7:12pm UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868/6 "2017-08-28T19:12:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.