Elasticsearch correlation getting failed

Elastic Stack Logstash
1

Hi,

I have request and response files which I need to process and correlate both the file fields and to correlate it. In my scenario filebeat picking up the files in random hence sometime my response files getting read instead of request files. So to cover all the scenarios I have the logic both in my response and request logstash configuration files. Below are my sample code.

Request section:

filter{
.......

elasticsearch {
			  hosts => ["xxxxx.xx:9200"]
			  query => "fileidres:%{fileidreq}"
			  fields => [
			        "status","status",
				"creationTime","creationTime",
				"fileidres","fileidres"
			       ]
}

if [status] == "CODE"
			{  <Some logic>
}
}

fileidres is the field created in response block and fileidreq was created in request block

Response Section:

 filter{
    grok {
    		  match => { "responseDesc" => "(?<status>CODE)" }  
    		}
    }

  1. I would like to know is thee any option to make Filebeat or Logstash to process the request files first before response files.

  2. Is there any attribute available in Elasticsearch filter to collect all data if the query is success.

Thanks in advance

2

Sorry, but this approach to merging events is fundamentally broken. You need to be able to synchronize so that the entries in one file is read before the other file's entries and Logstash has no mechanisms for that.

4

How can I synchronize the events?. Is my elasticsearch aggregation is correct?