Dead_lettter_queue plugin error and data processing behavior

Elastic Stack Logstash
1

Situation:

  • Logstash version (Docker Image 6.2.4)
  • Multi-Pipeline mode is used for executing many pipelines
  • dead-letter data processing pipeline is with id "main". We have 20+ pipelines apart from main
  • The idea of main pipeline is to process any dead-letter data generated by other 20+ pipelines into a separate indice that has no strict mappings defined
  • Example dead-letter picking configuration
    dead_letter_queue {
    path => "/var/log/logstash/deadletter"
    commit_offsets => true
    pipeline_id => "XYZ"
    }...this configuration is repeated for each other pipeline id.
  • Pipelines.yml representing multi-pipeline has a entry for deadletter pipeline with id as "main"

Below is the behavior observed, and any guidance on fixing the issue would be helpful

Logstash is started for the first time

  • deadletter/respective pipeline folder created
  • No dead letter queue data is processed though there is data in deadletter/.log files coming thru'
  • No error on docker logs around dlq plugin

Logstash is restarted (docker instance. NOTE: Only one instance is running)

  • Getting below error after each pipeline is started
    [ERROR] 2019-01-04 00:15:30.256 [[main]<dead_letter_queue] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
    Pipeline_id:main
    Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/var/log/logstash/deadletter", commit_offsets=>true, pipeline_id=>"REPEATS FOR EACH PIPELINE AFTER THE PIPELINE IS STARTED", id=>"b61022b1176319bb33e21c28df3dd9c244dc5c3157452e6d943c977559bfc862", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_df99ca06-9be8-463d-9013-78f87e16ada8", enable_metric=>true, charset=>"UTF-8">>
    Error: Invalid file. check version
    Exception: Java::JavaLang::RuntimeException
    Stack: org.logstash.common.io.RecordIOReader.(org/logstash/common/io/RecordIOReader.java:59)
    org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:126)
    org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97)
    org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82)
    java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
    org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
    org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
    RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60)
    usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther8:run(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514)
    usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.inputworker(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514)
    usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther0:inputworker(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)
    usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.block in start_input(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)
    org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
    org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
    java.lang.Thread.run(java/lang/Thread.java:748)

  • Pipeline processes dead letter log files (created by first start only)

Reference thread for getting dead-letter error data https://discuss.elastic.co/t/have-you-found-a-sollution-to-the-issue-with-de-dlq-obtain-reason/161076

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.