# Logstash persistent queue empty but all events coming in fine **URL:** \ **Category:** Logstash\ **Created:** [March 29, 2021, 1:47am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619 "2021-03-29T01:47:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Post date:** [March 29, 2021, 1:47am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/1 "2021-03-29T01:47:44Z")
I have configured logstash to use persistent queue. As per documentation, "When an input has events ready to process, it writes them to the queue." I was expecting to see some files in the folder '/Data/logstash/data/queue'. But were none during the whole run. There is nothing wrong in functionality. All the expected events did come in through. So I was wondering how the implementation is designed. Is it that only when current processing capacity overwhelmed that the files will be written in the queue folder? Or else by design every event has to be written on the queue before it is picked up by filters+outputs? If so I should have seen some files. Or is my configuration wrong? **Details** Version: 7.7.0 16 GB machine with 8GB allocated to Logstash via /etc/logstash/jvm.options Logstash.yml contents: ``` # ------------ Data path ------------------ path.data: /Data/logstash/data # ------------ Queuing Settings -------------- queue.type: persisted # path.queue: # queue.page_capacity: 64mb # queue.max_events: 0 queue.max_bytes: 300gb # queue.checkpoint.writes: 1024 # queue.checkpoint.interval: 1000 ``` ---
**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Post date:** [March 29, 2021, 8:30pm UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/2 "2021-03-29T20:30:10Z")
Guys any ideas? Should I reduce the RAM allocated to Logstash to just be able to see the files getting created? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 29, 2021, 10:48pm UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/3 "2021-03-29T22:48:15Z")
Hi @pk.241011 .... We're not all guys... 😉 Couple thoughts... You could probably just start Logstash with bad creds to Elasticsearch let it run a few minutes I would think that would work... You could create a Logstash user (which you really should anyways) let it run... Change the password in elasticsearch reload the Logstash config (break it simulate loss of connectivity ) then change it back... simulate reconnect. ---
**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Post date:** [March 30, 2021, 12:30am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/4 "2021-03-30T00:30:33Z")
Hi, Well here was the first attempt at creating a negative test condition. I put in wrong creds in Elasticsearch output of logstash. The output from logstash was: `[WARN] 2021-03-30 11:19:30.805 [Ruby-0-Thread-5: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://devops:xxxxxx@elasticinstance:9111/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticinstance:9111/'"}` And nothing still in the `/Data/logstash/data/queue` folder. And this time the data was lost. I see 401 error code. As per [documentation](https://www.elastic.co/guide/en/logstash/master/plugins-outputs-elasticsearch.html#_retry_policy) this should have been retried. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 30, 2021, 2:49am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/5 "2021-03-30T02:49:34Z")
Hi @pk.241011 How are you starting logstash. Perhaps you have a permission issues with the data dir. It should get initialized at startup. BTW that is a pretty big queue you have set up 300gb... perhaps just try the defaults first? I changed 1 line ``` # Internal queuing model, "memory" for legacy in-memory based queuing and # "persisted" for disk-based acked queueing. Defaults is memory # queue.type: persisted ``` I removed the logstash data directory to make sure I started clean I started a local Elasticsearch and ran this conf which reads the license file. ``` ################################## # Read License file ################################## input { file { path => "/Users/sbrown/workspace/elastic-install/7.11.2/logstash-7.11.2/LICENSE.txt" start_position => "beginning" sincedb_path => "/dev/null" } } output { # pump to stdout for debug stdout {codec => rubydebug} elasticsearch { hosts => ["localhost:9200"] index => "test-persistent-queue" } } ``` I used this command the -r means I can reload the conf and it will re-read the data / License File `sudo ./bin/logstash -r -f ./simple-file.conf` Logstash ran as expected and loaded 223 documents / lines The data/queue directory is initialized... if you are not seeing that that is a problem. Are there any errors in your logstash startup logs? How are you starting elasticsearch? ``` ceres:logstash-7.11.2 sbrown$ ls -lR data/queue/ total 0 drwxr-xr-x 6 root staff 192 Mar 29 19:35 main/ data/queue//main: total 128 -rw-r--r-- 1 root staff 34 Mar 29 19:35 checkpoint.head -rw-r--r-- 1 root staff 67108864 Mar 29 19:35 page.0 ceres:logstash-7.11.2 sbrown$ ``` BTW then I stopped Elasticsearch, then re-saved the conf file which cause the data to be re-read and logstash could not send.... ```auto [2021-03-29T19:40:17,785][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"} ``` Then the I re-started Elasticsearch did not touch logstash and once logstash reconnected the data was sent I had 446 documents `[2021-03-29T19:41:47,145][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}` ---
**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Post date:** [March 30, 2021, 4:11am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/6 "2021-03-30T04:11:49Z")
Yup you fixed it for me. 😀 It was permission issue. I was testing it on command line with root user. I should have been using the logstash user. Thanks for the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2021, 4:12am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619/7 "2021-04-27T04:12:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.