# Error in processing events from DLQ **URL:** \ **Category:** Logstash\ **Created:** [March 29, 2018, 7:02pm UTC](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149 "2018-03-29T19:02:50Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![pandeesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandeesh/32/29194_2.png) [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Post date:** [March 29, 2018, 7:02pm UTC](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149/1 "2018-03-29T19:02:50Z")
All, I am trying to expose the raw events in a index dedicated to DLQ. These are the fields I would like to expose: > index\_name (where it was unable to perform originally) > error\_reason(exception) > raw\_event(\_source) > document\_id Here's my current DLQ configuration: > input { > dead\_letter\_queue { > path =\> "/var/lib/logstash/dead\_letter\_queue" > commit\_offsets =\> true > } > } > output { > elasticsearch { > document\_type =\> "dlq" > codec =\> rubydebug { metadata =\> true } > hosts =\> ["host"] > index =\> "dlq-test-%{+YYYY.MM.DD}" > } } But the events are still failing to index with the same reason as original pipeline. I am getting: > [2018-03-29T11:55:18,463][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"dlq-test-2018.03.88", :\_type=\>"dlq", :\_routing=\>nil}, 2018-03-29T18:54:29.400Z xxx-prod-10-3-10-119 %{message}], :response=\>{"index"=\>{"\_index"=\>"dlq-test-2018.03.88", "\_type"=\>"dlq", "\_id"=\>"AWJzHTs9B5EI", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"object mapping for [fields.xxx.xxx]tried to parse field [properties] as object, but found a concrete value"}}}} even with codec as JSON, I am seeing the same behavior. Either with json or rubydebug codec, i am unable to see the error message/exception in the dlq-test\* index in kibana. if I want to only retrieve error, index\_name, document\_id and raw\_event, is this the correct codec to use? thanks for your help. ---
**Author:** ![pandeesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandeesh/32/29194_2.png) [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Post date:** [March 29, 2018, 7:40pm UTC](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149/2 "2018-03-29T19:40:38Z")
This looks like promising: [https://lukewaite.ca/posts/2017/10/13/viewing-logstash-dlq-in-kibana.html](https://lukewaite.ca/posts/2017/10/13/viewing-logstash-dlq-in-kibana.html) exploring this. thanks ---
**Author:** ![pandeesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandeesh/32/29194_2.png) [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Post date:** [March 29, 2018, 9:18pm UTC](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149/3 "2018-03-29T21:18:20Z")
However I have found another problem while implementing this. my dlq pipeline is similar to the one mentioned here:kewaite.ca/posts/2017/10/13/viewing-logstash-dlq-in-kibana.html > input { > dead\_letter\_queue { > path =\> "/var/lib/logstash/dead\_letter\_queue" > commit\_offsets =\> true > } > } > filter { > # First, we must capture the entire event, and write it to a new > # field; we'll call that field `failed_message` > ruby { > code =\> "event.set('failed\_message', event.to\_json())" > } > # Next, we prune every field off the event except for the one we've > # just created. Note that this does not prune event metadata. > prune { > whitelist\_names =\> ["^failed\_message$"] > } > # Next, convert the metadata timestamp to one we can parse with a > # date filter. Before conversion, this field is a Logstash::Timestamp. > # [http://www.rubydoc.info/gems/logstash-core/LogStash/Timestamp](http://www.rubydoc.info/gems/logstash-core/LogStash/Timestamp) > ruby { > code =\> "event.set('timestamp', event.get('[@metadata][dead\_letter\_queue][entry\_time]').toString())" > } > # Apply the date filter. > date { > match =\> ["timestamp", "ISO8601"] > } > # Pull useful information out of the event metadata provided by the dead > # letter queue, and add it to the new event. > mutate { > add\_field =\> { > "message" =\> "%{[@metadata][dead\_letter\_queue][reason]}" > "plugin\_id" =\> "%{[@metadata][dead\_letter\_queue][plugin\_id]}" > "plugin\_type" =\> "%{[@metadata][dead\_letter\_queue][plugin\_type]}" > } > } > } > output { > elasticsearch { > document\_type =\> "dlq" > hosts =\> ["host"] > index =\> "dlq-events-%{+YYYY.MM.dd}" > } } However, when there's no messages/events in the input dlq path, it breaks with , > [2018-03-29T14:13:57,569][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `toString' for nil:NilClass [2018-03-29T14:13:57,747][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:205:in `sprintf'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:168:in `event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:44:in `event\_action\_tuple'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'", "org/jruby/RubyArray.java:2414:in `map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi\_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output\_delegator.rb:49:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:434:in `output\_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:433:in `output\_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:381:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in `start\_workers'"]} Is there a recommended way to handle this case? It's obvious that dead\_letter\_queue can be often empty and it has data only sometimes when there are issues. Is there a technique to handle this scenario gracefully without failing all the logstash pipelines running? I feel, > commit\_offsets=\> true is not doing the right thing here. I expect it should not pass any records to filter. Upon, further investigation it looks like DLQ confg in primary pipeline is generating the single byte file with empty content which triggers this error in the DLQ pipeline. is it a known issue or any workaround for this? plug-in version: > logstash-input-dead\_letter\_queue (1.1.2) This is the dlq config in primary pipeline main: ``` # ------------ Dead-Letter Queue Settings -------------- # Flag to turn on dead-letter queue. # dead_letter_queue.enable: true dead_letter_queue.max_bytes: 4g ``` also, noticed that when I have just a single pipeline without the secondary one DLQ pipeline, it's able to write the proper logs to DLQ directory. while introducing pipelines.yml with 2 pipelines(primary and DLQ) which overrides the logstash.yml starts causing the single byte issue. Thanks for your help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 26, 2018, 9:18pm UTC](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149/4 "2018-04-26T21:18:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.