Enriching using Elasticsearch plugin , request and response message does not match if not enough delay between messages

Elastic Stack Logstash
1

I need to match response and request message so I decided to enrich response message with data from request.
I use elastic plugin

Code checks if message is response it finds request with same ID and enrich line with required values + calculate duration.

if [pair_frame_rq]  { 
            elasticsearch {
                    hosts => ["127.0.0.1:9200"]
                    index => "pcap-test"
                    query => "pair_id:%{pair_id} AND source_file:%{source_file}" 
                    fields => { "http_data" => "http_data_rq" }
                    fields => { "@timestamp" => "started" }
                    fields => { "msisdn" => "msisdn_rq" }
            }

The Enrichment works excellend if I upload 1 line by one with sufficent delay.
In case I insert whole file. The logstash is trying to execute elasticsearch request before that searched request messages is inserted . and response with error message

[2018-10-30T06:25:05,559][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"pcap-test", :query=>"pair_id:20_25 AND source_file:data1.json", :event=>#LogStash::Event:0x72a0a76e, :error=>#<Elasticsearch::Transport::Transport::Errors::NotFound: [404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"pcap-test","index_uuid":"na","index":"pcap-test"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"pcap-test","index_uuid":"na","index":"pcap-test"},"status":404}>}

If I repeat the same thing slowly (insert 1 line wait 4 seconds insert 2 line wait...) it works.
Any idea how to resolve it?
Is there any parameter , so logstash will not process the next request until the previous is correctly processed in Elasticsearch database in such a case I think it would work.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.