# How to extract json from the response field in the output of dead letter queue? **URL:** \ **Category:** Logstash\ **Created:** [August 15, 2019, 7:05am UTC](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306 "2019-08-15T07:05:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\ **Post date:** [August 15, 2019, 7:05am UTC](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306/1 "2019-08-15T07:05:31Z")
hi guys , my scripts is as follows :- ````auto input { dead_letter_queue { path => "${LS_HOME}/data/dead_letter_queue" pipeline_id => "main" } } filter { ruby { path => "${LS_HOME}/config/failed-index-create.rb" script_params => { "indexNamePrefix" => "xxx" } } prune { whitelist_names => ["^failed_message$"] } date { match => ["timestamp", "ISO8601"] } json { source => "failed_message" target => "failed_message_json" } mutate { add_field => { "reason" => "%{[@metadata][dead_letter_queue][reason]}" # "plugin_id" => "%{[@metadata][dead_letter_queue][plugin_id]}" # "plugin_type" => "%{[@metadata][dead_letter_queue][plugin_type]}" # } # } json { source => "%{[@metadata][dead_letter_queue][reason]}" target => "reason_json" #remove_char_key => ">\[\]" #remove_char_value => ">\[\]" #include_brackets => false } #ruby { #code => " # hash = event.to_hash # hash.each { |key,value| # if value.include? '\"' # event.set(key, value.gsub!('\"', '')) # end # } " #} } output { #elasticsearch{ # hosts => ["localhost:9200"] #index => "%{[@metadata][indexName]}" #} stdout { codec => rubydebug { metadata => true } } } in my case the output in reason field is :- ``` reason" : """Could not index event to Elasticsearch. status: 400, action: ["update", {:_id=>"eOxcjn4/cA9RYi55c12BK0yEXHs=", :_index=>"xxx-x-201x", :_type=>"_doc", :routing=>nil, :retry_on_conflict=>1}, #], **response** : {"update"=>{"_index"=>"xxx-x-201x", "_type"=>"_doc", "_id"=>"eOxcjn4/cA9RYi55c12BK0yEXHs=", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [billDueDate] of type [date] in document with id 'eOxcjn4/cA9RYi55c12BK0yEXHs='", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [abc] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}""" i want to extract the response array inside the reason field and index it into elasticsearch like this :- and the expected output that need to be extracted from the above json and index this way into elasticsearch :- { "update": { "_index": "xxx-x-201x", "_type": "_doc", "_id": "eOxcjn4/cA9RYi55c12BK0yEXHs=", "status": 400, "error": { "type": "mapper_parsing_exception", "reason": "failed to parse field [billDueDate] of type [date] in document with id 'eOxcjn4/cA9RYi55c12BK0yEXHs='", "caused_by": { "type": "illegal_argument_exception", "reason": "failed to parse date field [abc] with format [strict_date_optional_time||epoch_millis]", "caused_by": { "type": "date_time_parse_exception", "reason": "Failed to parse with all enclosed parsers" } } } } i want to parse this in the same dlq script. ```` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 16, 2019, 6:59pm UTC](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306/2 "2019-08-16T18:59:51Z")
``` mutate { gsub => ["reason", ".*\*\*response\*\* : ", ""] } mutate { gsub => ["reason", "=>", ":"] } mutate { gsub => ["reason", '"""$', ""] } json { source => "reason" } ``` will get you ``` "update" => { "error" => { "reason" => "failed to parse field [billDueDate] of type [date] in document with id 'eOxcjn4/cA9RYi55c12BK0yEXHs='", "type" => "mapper_parsing_exception", "caused_by" => { "reason" => "failed to parse date field [abc] with format [strict_date_optional_time||epoch_millis]", "type" => "illegal_argument_exception", "caused_by" => { "type" => "date_time_parse_exception", "reason" => "Failed to parse with all enclosed parsers" } } }, "_index" => "xxx-x-201x", "_type" => "_doc", "_id" => "eOxcjn4/cA9RYi55c12BK0yEXHs=", "status" => 400 }, ``` Using mutate to move the fields around to taste is left as an exercise for the reader. ---
**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\ **Post date:** [August 19, 2019, 12:01pm UTC](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306/3 "2019-08-19T12:01:39Z")
hi @Badger Thanks for the great answer , but it need to be tweaked a little bit. below has worked for me. ``` mutate { gsub => ["reason","^(.*)response: ",""]} mutate { gsub => ["reason", "=>", ":"] } ``` and rest of conversion in json string and mutating as per the requirement is done. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 16, 2019, 12:01pm UTC](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306/4 "2019-09-16T12:01:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.