# Elasticsearch for correlation

**URL:** <https://discuss.elastic.co/t/elasticsearch-for-correlation/241479>\
**Category:** Elasticsearch\
**Created:** [July 16, 2020, 2:13pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479 "2020-07-16T14:13:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Suat\_Bey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suat_bey/32/46322_2.png) [@Suat\_Bey](https://discuss.elastic.co/u/Suat_Bey)\
**Post date:** [July 16, 2020, 2:13pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/1 "2020-07-16T14:13:38Z")

</div>

Is it possible to write a query for elacticsearch something like this:

If same **IP** and same **username** generates 3 different **rule.id** within 15 min.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [July 16, 2020, 2:17pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/2 "2020-07-16T14:17:38Z")

</div>

That's the sort of thing we're aiming to make easier in future with [EQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/eql.html)

---

<div class="post-metadata">

**Author:** ![Suat\_Bey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suat_bey/32/46322_2.png) [@Suat\_Bey](https://discuss.elastic.co/u/Suat_Bey)\
**Post date:** [July 16, 2020, 2:21pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/3 "2020-07-16T14:21:17Z")

</div>

Thanks Mark, what about getting the same query with given **IP,username and rule.id** manually?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 16, 2020, 2:25pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/4 "2020-07-16T14:25:59Z")

</div>

You could use a composite aggregation to `group_by` using `date_histogram` (`15m`), and `terms` (`IP`, `username`) and aggregate via `cardinality` on `rule.id`. This would give you the results, but you still need to scan them to report if they breach `3`.

More advanced would be the same as above as transform. A transform will write the results to an index and you can use alerting on it (e.g. watcher).

++ if you don't care about results `<3`, you can use an ingest pipeline (as part of the transform destination) to drop those.

I do not have a fully fitting example, but [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-examples.html#example-clientips) might be a starting pointer. In the `group_by` its possible to add `date_histogram` regarding the `15m` time window you are looking for.

---

<div class="post-metadata">

**Author:** ![Suat\_Bey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suat_bey/32/46322_2.png) [@Suat\_Bey](https://discuss.elastic.co/u/Suat_Bey)\
**Post date:** [July 16, 2020, 2:35pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/5 "2020-07-16T14:35:19Z")

</div>

Is there any simple example that I can take a look ?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 16, 2020, 2:40pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/6 "2020-07-16T14:40:26Z")

</div>

I added a link after editing, sorry:

[https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-examples.html#example-clientips](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-examples.html#example-clientips)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 13, 2020, 2:40pm UTC](https://discuss.elastic.co/t/elasticsearch-for-correlation/241479/7 "2020-08-13T14:40:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.