How to Co-relate two document in the same index?

Elastic Stack Elasticsearch
1

Hi all,
I want to co-relate two documents. Both the documents resides in same elastic index. My goal is to achive something like, if my logs says system is up but there is not system down in last one hour then it is good i dont need to worry about it. But if logs says system is up and there is system is down before 5 minutes then i need to raise some alarm. For example below are the two documents in my elastic.

	DOCUMENT-1
	{
      
      "@timestamp": "2018-07-18T**10:06:47.968-04:00**",
      "event": "XYZ system is down",
      ......
    }
	
	DOCUMENT-2
	{
      
      "@timestamp": "2018-07-18T**11:06:47.968-04:00**",
      "event": "XYZ system up",
      ......
    }

In case there is

Document-2(system is up) with timestamp > Document-1's timestamp (System is down) && timedifference between both the document is less then 30 minutees

then I want to co-relate two document and get both of them using elastic query language. Can some one guide me about the approach which I should take.

2

I don't think there is any easy way of doing so.
Apart may be sort by date any document related to XYZ and do the processing by yourself.

I'd recommend in such use case to do the computation at index time and store documents like:

{
"stopTime": "2018-07-18T**10:06:47.968-04:00**",  
"startTime": "2018-07-18T**11:06:47.968-04:00**"
}

You can use something like the aggregate filter in logstash for that.

I'm not sure otherwise if the pipeline aggregation could help in such a case. May be someone else has other ideas though.

3

Thank you david, for quick reply will think as you suggested.:grinning: Anyone has other idea?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.