# Enriching using Elasticsearch plugin , request and response message does not match if not enough delay between messages

**URL:** <https://discuss.elastic.co/t/enriching-using-elasticsearch-plugin-request-and-response-message-does-not-match-if-not-enough-delay-between-messages/154564>\
**Category:** Logstash\
**Created:** [October 30, 2018, 5:42am UTC](https://discuss.elastic.co/t/enriching-using-elasticsearch-plugin-request-and-response-message-does-not-match-if-not-enough-delay-between-messages/154564 "2018-10-30T05:42:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [October 30, 2018, 5:42am UTC](https://discuss.elastic.co/t/enriching-using-elasticsearch-plugin-request-and-response-message-does-not-match-if-not-enough-delay-between-messages/154564/1 "2018-10-30T05:42:32Z")

</div>

I need to match response and request message so I decided to enrich response message with data from request.  
I use [elastic plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)

Code checks if message is response it finds request with same ID and enrich line with required values + calculate duration.

> ```
> if [pair_frame_rq] { 
> elasticsearch {
> hosts => ["127.0.0.1:9200"]
> index => "pcap-test"
> query => "pair_id:%{pair_id} AND source_file:%{source_file}" 
> fields => { "http_data" => "http_data_rq" }
> fields => { "@timestamp" => "started" }
> fields => { "msisdn" => "msisdn_rq" }
> }
> 
> ```

The Enrichment works excellend if I upload 1 line by one with sufficent delay.  
In case I insert whole file. The logstash is trying to execute elasticsearch request before that searched request messages is inserted . and response with error message

> [2018-10-30T06:25:05,559][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=\>"pcap-test", :query=\>"pair\_id:20\_25 AND source\_file:data1.json", :event=\>#LogStash::Event:0x72a0a76e, :error=\>#\<Elasticsearch::Transport::Transport::Errors::NotFound: [404] {"error":{"root\_cause":[{"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_or\_alias","resource.id":"pcap-test","index\_uuid":"_na_","index":"pcap-test"}],"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_or\_alias","resource.id":"pcap-test","index\_uuid":"_na_","index":"pcap-test"},"status":404}\>}

If I repeat the same thing slowly (insert 1 line wait 4 seconds insert 2 line wait...) it works.  
Any idea how to resolve it?  
Is there any parameter , so logstash will not process the next request until the previous is correctly processed in Elasticsearch database in such a case I think it would work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 27, 2018, 5:42am UTC](https://discuss.elastic.co/t/enriching-using-elasticsearch-plugin-request-and-response-message-does-not-match-if-not-enough-delay-between-messages/154564/2 "2018-11-27T05:42:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.