# How to correlate events in ES

**URL:** <https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307>\
**Category:** Elasticsearch\
**Created:** [May 26, 2017, 10:48pm UTC](https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307 "2017-05-26T22:48:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\
**Post date:** [May 26, 2017, 10:48pm UTC](https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307/1 "2017-05-26T22:48:39Z")

</div>

I want to correlate events in IIS and IDS logs so that I can get the client IPs that exist in both log types in which each matches certain conditions (hits admin.aspx with 200 response code in IIS and generates critical log level in IDS).

Sample logs

```auto
{
  "src_ip": "10.0.0.1",
  "http_status": 200,
  "path": "/admin.aspx",
  "type": "iis"
},
{
  "src_ip": "10.0.0.2",
  "http_status": 200,
  "path": "/admin.aspx",
  "type": "iis"
},
{
  "src_ip": "10.0.0.1",
  "log_level": "critical",
  "type": "ids"
},
{
  "src_ip": "10.0.0.2",
  "log_level": "info",
  "type": "ids"
}

```

Can we write a query/agg in ES that returns only `10.0.0.1` which matches the condition I specify?

---

<div class="post-metadata">

**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\
**Post date:** [June 3, 2017, 8:02am UTC](https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307/2 "2017-06-03T08:02:15Z")

</div>

Anyone knows how to write this query?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [June 3, 2017, 10:30am UTC](https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307/3 "2017-06-03T10:30:14Z")

</div>

This is what I've used in python before to merge content in two indices. Note it streams ALL docs.

```
mergeQuery={
   "query": {
      "match_all": { }
   },
   "sort": [
      {
         "src_ip": {
            "order": "asc"
         }
      }
   ]
}

for doc in helpers.scan(es,
                    index="indexA,indexB".
                    query=mergeQuery,
                    size=args.readsPerBulk,
                    scroll=args.maxTimeToProcessScrollPage,
                    preserve_order=True):
                    
    # Do whatever logic here to spot consecutive docs
    # with same key and merge
```

---

<div class="post-metadata">

**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\
**Post date:** [June 13, 2017, 9:34pm UTC](https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307/4 "2017-06-13T21:34:26Z")

</div>

Basically, I can accomplish this by retrieving data from ES and process. I just wonder if there're any agg or query types for similar use cases.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 11, 2017, 9:35pm UTC](https://discuss.elastic.co/t/how-to-correlate-events-in-es/87307/5 "2017-07-11T21:35:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.