We are using logstash 7.4.2, we faced queue corruption issue, below error.
We tried pqcheck and pqrepair logstash utility however its not fixed.
Any pointer would be really helpful.
Thank you.
Error :[2020-05-08T11:05:54,891][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Page file size is too small to hold elements
at org.logstash.ackedqueue.io.MmapPageIOV2.mapFile(MmapPageIOV2.java:275) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.io.MmapPageIOV2.recover(MmapPageIOV2.java:134) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.Queue.open(Queue.java:210) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:101) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:42) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:39) ~[logstash-core.jar:?]
at org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:171) [logstash-core.jar:?]
at org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:354) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:143) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:105) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:92) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.RubyClass.newInstance(RubyClass.java:915) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:540) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:204) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:191) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208) [jruby-complete-9.2.8.0.jar:?]
at home.elysium.installs.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/home/elysium/installs/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:326) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?]
pqcheck output :
Exception in thread "main" java.lang.NumberFormatException: For input string: "head.tmp"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.logstash.ackedqueue.PqCheck.cpNum(PqCheck.java:79)
at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
at java.util.TimSort.sort(TimSort.java:220)
at java.util.Arrays.parallelSort(Arrays.java:1113)
at java.util.stream.SortedOps$OfRef.opEvaluateParallel(SortedOps.java:158)
at java.util.stream.AbstractPipeline.opEvaluateParallelLazy(AbstractPipeline.java:705)
at java.util.stream.AbstractPipeline.sourceSpliterator(AbstractPipeline.java:432)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.logstash.ackedqueue.PqCheck.checkPQ(PqCheck.java:45)
at org.logstash.ackedqueue.PqCheck.main(PqCheck.java:28)