Deadletter queue

Elastic Stack Logstash
1 
Hi 

have to add event,metadata in dead letter queue when will get any exception in http filter.

am using pipelines.yml as follows
=================================================================
- pipeline.id: pipeline21
  path.config: "/etc/logstash/conf.d/ds_test1conf"
  queue.type: persisted
- pipeline.id: pipeline21
  path.config: "/etc/logstash/conf.d/ds_test4.conf"
  queue.type: persisted
  dead_letter_queue.enable: true
=================================================================
ds_test4.conf file 
input {
kafka {
    bootstrap_servers => "172.20.188.11:9092,172.20.188.31:9092,172.20.188.30:9092"
    topics => ["DS_Test"]
    decorate_events => true
    group_id => "ds_test"
    codec => json
    max_poll_records => "100"
    consumer_threads => 2
    session_timeout_ms => "30000"
    auto_offset_reset => "earliest"
    client_id => "logstash-ingestion"
 }
}
filter
{
        mutate { add_field => { "[@metadata][document_id]" => "36_20006_7"}}
        http {
                  body_format => "json"
                  follow_redirects => false
                  body => {"Record" => [{"nounModifier"=>"" "itemId"=>"20006" "field"=>"itemDescription" "normalizedText"=>"" "softDelete"=>"pank1" "IndexedOn"=>"" "appId"=>"36" "contactcode"=>"0" "text"=>"ball 20 mm" "id"=>"36_20006_7" "type"=>"AS"}]}
                  url => "https://dsmdmdev-itemmdm.seadevds.p.azurewebsites.net/normalizer/update_es"
                  verb => "POST"
                  headers => {"Content-Type" => "application/json"}
           }
		   if [tags][0] == "_httprequestfailure" {
		    mutate { add_field => { "[@metadata][code]" => "pankaj"}}
		   }
}

output{
elasticsearch {
                                hosts => "http://172.20.188.15:9200"
                                user => "elastic"
                                password => "changeme"
                                index => "flowersfoods_62"
                                document_id => "test"
                                #document_id => "12_100"
                                action => "index"
                }
        stdout { codec => rubydebug { metadata => true} }
}
===================================================================
inside tags am getting "_httprequestfailure " so i want to add this event in "deadletterqueue". but cant able to see anything in deadletterqueue folder in logstash. but same event getting index in elastic. but as per my requirement if any exception is there we needs to add event inside deadletterqueue.
2

That is not how a DLQ works. The elasticsearch output sends an event to the DLQ if it is unable to send it to elasticsearch.

You could use a conditional in the output section to send events that have an _httprequestfailure tag to a different output, but that output could not be a DLQ.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.