Logstash 5.6 Persistent Queue issue (Docker)


(Stevens J Michel) #1

Hello,
I currently have logstash running within a docker container and when I have it configured for persistent queue it works as it should. The only issue is when I mount a volume to where the persistent queue directory resides in the vm the logstash pipeline will not start. It seems as if the paging mechinism will not work if you mount a volume to where the queues exist and it sucks because if the docker VM has a critical failure then events that were saved to the persistent queues are lost! Thus the reason why having it mount to a volume on the host is essential. looks like the offended class is: JrubyAckedQueueExtLibrary.java. Anyone else ran into this type of issue? And FYI- Mounting a volume to the dead_letter_queue works perfectly fine, the .sincedb files and queue files get copied to the host with no exceptions.

logstash.yml
queue.type: persisted
path.queue: /opt/logstash/persistance/data
queue.max_events: 0
queue.max_bytes: 10240mb
queue.checkpoint.acks: 1024
queue.checkpoint.writes: 1024
queue.checkpoint.interval: 1000

Logstash Exception
[2017-09-19T20:07:33,452][ERROR][logstash.pipeline ] Logstash failed to create queue {"exception"=>"Invalid argument", "backtrace"=>["org/logstash/ackedqueue/ext/JrubyAckedQueueExtLibrary.java:133:in open'", "/opt/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:41:inwith_queue'", "/opt/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:30:in create_file_based'", "/opt/logstash/logstash-core/lib/logstash/queue_factory.rb:29:increate'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:168:in initialize'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:286:increate_pipeline'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:95:in register_pipeline'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:309:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:204:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/opt/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}
[2017-09-19T20:07:33,458][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Invalid argument"}
[


(Magnus B├Ąck) #2

What Docker storage driver are you using? What kind of backing file system? Which version of Docker? Which kernel?

Possibly related:


(Stevens J Michel) #3

Thanks for the response. Looks like I figured it out. Seems there is a kernel conflict with mounting on a mac host. mounting on a linux vm has no issues :slight_smile:


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.