Unable to restart LS with: Logstash failed to create queue

Hi,
We have Logstash 7.5.1 configure with pipelines, our pipelines configuration is:

- pipeline.id: main
  path.config: "/etc/logstash/config/main/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1
- pipeline.id: to_filesystem
  path.config: "/etc/logstash/config/to_filesystem/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1
- pipeline.id: to_syslog
  path.config: "/etc/logstash/config/to_syslog/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1
- pipeline.id: to_qradar
  path.config: "/etc/logstash/config/to_qradar/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1  

Everything was working fine until today we noticed Logstash was down, and when we try to start it up, it fails. In the logstash log we have messages with:

[2020-04-01T16:00:44,444][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Unable to allocate 1015543052 more bytes for persisted queue on top of its current usage of 58198772 bytes
...........
[2020-04-01T16:00:44,476][DEBUG][org.logstash.ackedqueue.io.MmapPageIOV2] PageIO recovery element index:46319, readNextElement exception: Element seqNum 0 is expected to be 1329076
[2020-04-01T16:00:44,499][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Unable to allocate 1015543052 more bytes for persisted queue on top of its current usage of 58198772 bytes
........
[2020-04-01T16:00:44,529][DEBUG][org.logstash.ackedqueue.io.MmapPageIOV2] PageIO recovery element index:46319, readNextElement exception: Element seqNum 0 is expected to be 1416639
[2020-04-01T16:00:44,531][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Unable to allocate 1015543052 more bytes for persisted queue on top of its current usage of 58198772 bytes
.........
[2020-04-01T16:00:44,598][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<to_qradar>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}

Can anyone help? I was thinking of deleting the queues and try to start, but as this is a Testing server, we would like to know what is happening to avoit it to happen on production.

thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.