Unable to restart LS with: Logstash failed to create queue

Elastic Stack Logstash
1

Hi,
We have Logstash 7.5.1 configure with pipelines, our pipelines configuration is:

- pipeline.id: main
  path.config: "/etc/logstash/config/main/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1
- pipeline.id: to_filesystem
  path.config: "/etc/logstash/config/to_filesystem/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1
- pipeline.id: to_syslog
  path.config: "/etc/logstash/config/to_syslog/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1
- pipeline.id: to_qradar
  path.config: "/etc/logstash/config/to_qradar/*.conf"
  queue.type: persisted
  queue.checkpoint.writes: 1
  pipeline.workers: 1

Everything was working fine until today we noticed Logstash was down, and when we try to start it up, it fails. In the logstash log we have messages with:

[2020-04-01T16:00:44,444][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Unable to allocate 1015543052 more bytes for persisted queue on top of its current usage of 58198772 bytes
...........
[2020-04-01T16:00:44,476][DEBUG][org.logstash.ackedqueue.io.MmapPageIOV2] PageIO recovery element index:46319, readNextElement exception: Element seqNum 0 is expected to be 1329076
[2020-04-01T16:00:44,499][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Unable to allocate 1015543052 more bytes for persisted queue on top of its current usage of 58198772 bytes
........
[2020-04-01T16:00:44,529][DEBUG][org.logstash.ackedqueue.io.MmapPageIOV2] PageIO recovery element index:46319, readNextElement exception: Element seqNum 0 is expected to be 1416639
[2020-04-01T16:00:44,531][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Unable to allocate 1015543052 more bytes for persisted queue on top of its current usage of 58198772 bytes
.........
[2020-04-01T16:00:44,598][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<to_qradar>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}

Can anyone help? I was thinking of deleting the queues and try to start, but as this is a Testing server, we would like to know what is happening to avoit it to happen on production.

thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.