Dead_lettter_queue plugin error and data processing behavior


  • Logstash version (Docker Image 6.2.4)
  • Multi-Pipeline mode is used for executing many pipelines
  • dead-letter data processing pipeline is with id "main". We have 20+ pipelines apart from main
  • The idea of main pipeline is to process any dead-letter data generated by other 20+ pipelines into a separate indice that has no strict mappings defined
  • Example dead-letter picking configuration
    dead_letter_queue {
    path => "/var/log/logstash/deadletter"
    commit_offsets => true
    pipeline_id => "XYZ"
    }...this configuration is repeated for each other pipeline id.
  • Pipelines.yml representing multi-pipeline has a entry for deadletter pipeline with id as "main"

Below is the behavior observed, and any guidance on fixing the issue would be helpful

Logstash is started for the first time

  • deadletter/respective pipeline folder created
  • No dead letter queue data is processed though there is data in deadletter/.log files coming thru'
  • No error on docker logs around dlq plugin

Logstash is restarted (docker instance. NOTE: Only one instance is running)

  • Getting below error after each pipeline is started
    [ERROR] 2019-01-04 00:15:30.256 [[main]<dead_letter_queue] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
    Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/var/log/logstash/deadletter", commit_offsets=>true, pipeline_id=>"REPEATS FOR EACH PIPELINE AFTER THE PIPELINE IS STARTED", id=>"b61022b1176319bb33e21c28df3dd9c244dc5c3157452e6d943c977559bfc862", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_df99ca06-9be8-463d-9013-78f87e16ada8", enable_metric=>true, charset=>"UTF-8">>
    Error: Invalid file. check version
    Exception: Java::JavaLang::RuntimeException
    usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.block in start_input(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)

  • Pipeline processes dead letter log files (created by first start only)

Reference thread for getting dead-letter error data

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.