I have two events: test_start and test_end
When I receive test_start event, I clone one more event from it test_attempt and index it in elasticsearch.
Then when test_end event arrives, I extract document id and index for associated test_attempt index and update that document with data generated from test_end event.
Below is the pipeline configuration:
41
42 mutate {
43 add_field => {
44 "testEndTime" => "%{event_time}"
45 }
46 }
47
48 elasticsearch {
49 hosts => ["172.31.2.218:9200", "172.31.2.208:9200", "172.31.1.29:9200"]
50 index => "service-logs-*"
51 query => "session.connId:1234 AND derived_event:TEST_ATTEMPT"
52 result_size => 1
53 docinfo_fields => {
54 "_id" => "[@metadata][doc_id]"
55 "_index" => "[@metadata][doc_index]"
56 }
57 }
58 }```
Output configuration
```# update existing document if derived event is TEST_ATTEMPT and session event is TEST_END
66 if [derived_event] == "TEST_ATTEMPT" and [session][event] == "TEST_END" {
67 elasticsearch {
68 hosts => ["172.31.2.218:9200", "172.31.2.208:9200", "172.31.1.29:9200"]
69 index => "%{[@metadata][doc_index]}"
70 document_id => "%{[@metadata][doc_id]}"
71 action => "update"
72 }
73 }else {
74 # create new document in elasticserach
75 elasticsearch {
76 hosts => ["172.31.2.218:9200", "172.31.2.208:9200", "172.31.1.29:9200"]
77 index => "service-logs-%{do_service}-%{+YYYY.MM.dd}"
78 action => "create"
79 data_stream => "false"
80 ilm_enabled => "false"
81 }
82 }```
Now when both events are triggered with a negligible gap between them, ES throws 404 as it can not find the document.
So, I am trying to enable DLQ and then reprocess it after some delay.
But I am not able to turn on DLQ.
I have updated the config:
`dead_letter_queue.enable: true`
After turning on DLQ, I can see a directory created for dead_letter_queue, but there are no files created inside, even after elasticsearch output plugin throws 404.
Also when I do `/_cat/indices`, I can see
%{[@metadata][doc_index]}
This gets created when the elasticsearch filter lookup fails. Output responds with 404, but this index gets created.
What am I doing wrong?
logstash version: 7.14