# DLQ not enabled for logstasha **URL:** \ **Category:** Logstash\ **Created:** [August 10, 2021, 2:56pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940 "2021-08-10T14:56:08Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Krrish\_Raj1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krrish_raj1/32/77329_2.png) [@Krrish\_Raj1](https://discuss.elastic.co/u/Krrish_Raj1)\ **Post date:** [August 10, 2021, 2:56pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/1 "2021-08-10T14:56:08Z")
I have two events: `test_start and test_end` When I receive test\_start event, I clone one more event from it `test_attempt` and index it in elasticsearch. Then when test\_end event arrives, I extract document id and index for associated test\_attempt index and update that document with data generated from test\_end event. Below is the pipeline configuration: ````auto 41 42 mutate { 43 add_field => { 44 "testEndTime" => "%{event_time}" 45 } 46 } 47 48 elasticsearch { 49 hosts => ["172.31.2.218:9200", "172.31.2.208:9200", "172.31.1.29:9200"] 50 index => "service-logs-*" 51 query => "session.connId:1234 AND derived_event:TEST_ATTEMPT" 52 result_size => 1 53 docinfo_fields => { 54 "_id" => "[@metadata][doc_id]" 55 "_index" => "[@metadata][doc_index]" 56 } 57 } 58 }``` Output configuration ```# update existing document if derived event is TEST_ATTEMPT and session event is TEST_END 66 if [derived_event] == "TEST_ATTEMPT" and [session][event] == "TEST_END" { 67 elasticsearch { 68 hosts => ["172.31.2.218:9200", "172.31.2.208:9200", "172.31.1.29:9200"] 69 index => "%{[@metadata][doc_index]}" 70 document_id => "%{[@metadata][doc_id]}" 71 action => "update" 72 } 73 }else { 74 # create new document in elasticserach 75 elasticsearch { 76 hosts => ["172.31.2.218:9200", "172.31.2.208:9200", "172.31.1.29:9200"] 77 index => "service-logs-%{do_service}-%{+YYYY.MM.dd}" 78 action => "create" 79 data_stream => "false" 80 ilm_enabled => "false" 81 } 82 }``` Now when both events are triggered with a negligible gap between them, ES throws 404 as it can not find the document. So, I am trying to enable DLQ and then reprocess it after some delay. But I am not able to turn on DLQ. I have updated the config: `dead_letter_queue.enable: true` After turning on DLQ, I can see a directory created for dead_letter_queue, but there are no files created inside, even after elasticsearch output plugin throws 404. Also when I do `/_cat/indices`, I can see %{[@metadata][doc_index]} This gets created when the elasticsearch filter lookup fails. Output responds with 404, but this index gets created. What am I doing wrong? logstash version: 7.14 ```` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2021, 4:10pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/2 "2021-08-10T16:10:47Z")
What exactly gets the 404 error? Is it the elasticsearch filter? If the elasticsearch output, is it the bullk request or one of the documents within it? ---
**Author:** ![Krrish\_Raj1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krrish_raj1/32/77329_2.png) [@Krrish\_Raj1](https://discuss.elastic.co/u/Krrish_Raj1)\ **Post date:** [August 10, 2021, 6:02pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/3 "2021-08-10T18:02:07Z")
The output plugin throws 404 error. It basically complains that document does not exists. When the lookup fails in elastic filter plugin, the metadata field is not pupulated with index name and doc id. And I think that should be the reason for throwing 404. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2021, 6:09pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/4 "2021-08-10T18:09:53Z")
You need to distinguish whether the \_bulk request is getting a 404 response or a document within it is getting a 404 response. The latter goes to the DLQ, the former does not. ---
**Author:** ![Krrish\_Raj1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krrish_raj1/32/77329_2.png) [@Krrish\_Raj1](https://discuss.elastic.co/u/Krrish_Raj1)\ **Post date:** [August 10, 2021, 7:59pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/5 "2021-08-10T19:59:24Z")
It looks like a bulk request to me. [WARN] 2021-08-10 19:58:16.339 [[main]\>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=\>404, :action=\>["update", {:\_id=\>"%{[@metadata][doc\_id]}", :\_index=\>"%{[@metadata][doc\_index]}", :routing=\>nil, :retry\_on\_conflict=\>1}, {"message"=\>"[connector] AMQP connecting.....", "event\_time"=\>"2021-08-10T01:25:07.125Z", "session"=\>{"event"=\>"TEST\_END", "connId"=\>"1234"}, "do\_service"=\>"ds-websocket", "derived\_event"=\>"TEST\_ATTEMPT", "source\_ip"=\>"172.31.1.155", "level"=\>"info", "type"=\>"derived", "@timestamp"=\>2021-08-10T19:57:54.222Z, "@version"=\>"1", "testEndTime"=\>"2021-08-10T01:25:07.125Z"}], :response=\>{"update"=\>{"\_index"=\>"%{[@metadata][doc\_index]}", "\_type"=\>"\_doc", "\_id"=\>"%{[@metadata][doc\_id]}", "status"=\>404, "error"=\>{"type"=\>"document\_missing\_exception", "reason"=\>"[\_doc][%{[@metadata][doc\_id]}]: document missing", "index\_uuid"=\>"lzVTEX1SQQKOmjiiRe7mJA", "shard"=\>"0", "index"=\>"%{[@metadata][doc\_index]}"}}}} When I turn on debug log, I can see: [DEBUG] 2021-08-10 20:08:54.686 [[main]\>worker0] DefaultManagedHttpClientConnection - http-outgoing-3: set socket timeout to 60000 [DEBUG] 2021-08-10 20:08:54.686 [[main]\>worker0] MainClientExec - Executing request POST /\_bulk HTTP/1.1 [DEBUG] 2021-08-10 20:08:54.687 [[main]\>worker0] MainClientExec - Target auth state: UNCHALLENGED [DEBUG] 2021-08-10 20:08:54.687 [[main]\>worker0] MainClientExec - Proxy auth state: UNCHALLENGED [DEBUG] 2021-08-10 20:08:54.687 [[main]\>worker0] headers - http-outgoing-3 \>\> POST /\_bulk HTTP/1.1 [DEBUG] 2021-08-10 20:08:54.687 [[main]\>worker0] headers - http-outgoing-3 \>\> Connection: Keep-Alive [DEBUG] 2021-08-10 20:08:54.687 [[main]\>worker0] headers - http-outgoing-3 \>\> Content-Type: application/json [DEBUG] 2021-08-10 20:08:54.687 [[main]\>worker0] headers - http-outgoing-3 \>\> Content-Length: 474 [DEBUG] 2021-08-10 20:08:54.688 [[main]\>worker0] headers - http-outgoing-3 \>\> Host: 172.31.2.208:9200 [DEBUG] 2021-08-10 20:08:54.688 [[main]\>worker0] headers - http-outgoing-3 \>\> User-Agent: Manticore 0.7.0 [DEBUG] 2021-08-10 20:08:54.688 [[main]\>worker0] headers - http-outgoing-3 \>\> Accept-Encoding: gzip,deflate [DEBUG] 2021-08-10 20:08:54.689 [[main]\>worker0] wire - http-outgoing-3 \>\> "POST /\_bulk HTTP/1.1[\r][\n]" [DEBUG] 2021-08-10 20:08:54.690 [[main]\>worker0] wire - http-outgoing-3 \>\> "Connection: Keep-Alive[\r][\n]" [DEBUG] 2021-08-10 20:08:54.690 [[main]\>worker0] wire - http-outgoing-3 \>\> "Content-Type: application/json[\r][\n]" [DEBUG] 2021-08-10 20:08:54.690 [[main]\>worker0] wire - http-outgoing-3 \>\> "Content-Length: 474[\r][\n]" [DEBUG] 2021-08-10 20:08:54.690 [[main]\>worker0] wire - http-outgoing-3 \>\> "Host: 172.31.2.208:9200[\r][\n]" [DEBUG] 2021-08-10 20:08:54.690 [[main]\>worker0] wire - http-outgoing-3 \>\> "User-Agent: Manticore 0.7.0[\r][\n]" [DEBUG] 2021-08-10 20:08:54.691 [[main]\>worker0] wire - http-outgoing-3 \>\> "Accept-Encoding: gzip,deflate[\r][\n]" [DEBUG] 2021-08-10 20:08:54.691 [[main]\>worker0] wire - http-outgoing-3 \>\> "[\r][\n]" [DEBUG] 2021-08-10 20:08:54.691 [[main]\>worker0] wire - http-outgoing-3 \>\> "{"update":{"\_id":"%{[@metadata][doc\_id]}","\_index":"%{[@metadata][doc\_index]}","routing":null,"retry\_on\_conflict":1}}[\n]" [DEBUG] 2021-08-10 20:08:54.691 [[main]\>worker0] wire - http-outgoing-3 \>\> "{"doc":{"message":"[connector] AMQP connecting.....","@version":"1","do\_service":"ds-websocket","event\_time":"2021-08-10T01:25:07.125Z","type":"derived","source\_ip":"172.31.1.155","derived\_event":"TEST\_ATTEMPT","@timestamp":"2021-08-10T20:08:31.340Z","level":"info","testEndTime":"2021-08-10T01:25:07.125Z","session":{"event":"TEST\_END","connId":"1234"}}}[\n]" [DEBUG] 2021-08-10 20:08:54.701 [[main]\>worker0] wire - http-outgoing-3 \<\< "HTTP/1.1 200 OK[\r][\n]" [DEBUG] 2021-08-10 20:08:54.702 [[main]\>worker0] wire - http-outgoing-3 \<\< "content-type: application/json; charset=UTF-8[\r][\n]" [DEBUG] 2021-08-10 20:08:54.702 [[main]\>worker0] wire - http-outgoing-3 \<\< "content-encoding: gzip[\r][\n]" [DEBUG] 2021-08-10 20:08:54.702 [[main]\>worker0] wire - http-outgoing-3 \<\< "content-length: 223[\r][\n]" [DEBUG] 2021-08-10 20:08:54.702 [[main]\>worker0] wire - http-outgoing-3 \<\< "[\r][\n]" ---
**Author:** ![Krrish\_Raj1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krrish_raj1/32/77329_2.png) [@Krrish\_Raj1](https://discuss.elastic.co/u/Krrish_Raj1)\ **Post date:** [August 10, 2021, 8:19pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/6 "2021-08-10T20:19:08Z")
How to handle failed requests in case of \_bulk API? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2021, 8:34pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/7 "2021-08-10T20:34:04Z")
> [@Krrish\_Raj1](#): > > Could not index event to Elasticsearch That is [here](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/476097712b67e0452099769b2c8b0ef6e63e40eb/lib/logstash/plugin_mixins/elasticsearch/common.rb#L263) in the code. It is a response for a document within the bulk request, and it is a response that is sent for DLQ processing. However, looking at [that processing](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/476097712b67e0452099769b2c8b0ef6e63e40eb/lib/logstash/plugin_mixins/elasticsearch/common.rb#L202), if the DLQ is enabled then it is written to it, otherwise it logs the message that you see logged. logstash does not think you have DLQ enabled. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2021, 8:36pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/8 "2021-08-10T20:36:15Z")
If the call to \_bulk fails it is retried forever. This was recently changed to exclude status 413 (content too large) since that will continue failing forever, so it effectively stops logstash processing anything. There are other cases where the retried call will fail forever, but I cannot think of them right now. ---
**Author:** ![Krrish\_Raj1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krrish_raj1/32/77329_2.png) [@Krrish\_Raj1](https://discuss.elastic.co/u/Krrish_Raj1)\ **Post date:** [August 10, 2021, 9:15pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/9 "2021-08-10T21:15:36Z")
Thanks for pointing out the code blocks. I will dig into it. But why I see %{[@metadata][doc\_index]} index created when I use /\_cat/indices. It is an empty index though, but if the variable is empty, shouldn't it just evaluate to null rather than the literal string? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2021, 9:33pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/10 "2021-08-10T21:33:28Z")
> [@Krrish\_Raj1](#): > > shouldn't it just evaluate to null rather than the literal string? No, the literal string is expected. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 7, 2021, 9:34pm UTC](https://discuss.elastic.co/t/dlq-not-enabled-for-logstasha/280940/11 "2021-09-07T21:34:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.