# Differences between 2 groups of documents by multiple criteria

**URL:** https://discuss.elastic.co/t/differences-between-2-groups-of-documents-by-multiple-criteria/237300
**Category:** Elasticsearch
**Created:** [June 16, 2020, 12:55pm UTC](https://discuss.elastic.co/t/differences-between-2-groups-of-documents-by-multiple-criteria/237300 "2020-06-16T12:55:54Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![mihailo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mihailo/32/45680_2.png) [@mihailo](https://discuss.elastic.co/u/mihailo)
#### Post date: [June 16, 2020, 12:55pm UTC](https://discuss.elastic.co/t/differences-between-2-groups-of-documents-by-multiple-criteria/237300/1 "2020-06-16T12:55:55Z")

</div>

Hello everyone,

Please take a look at this dummy documents:

```
[
    {
        "project": "a",
        "run": 1,
        "tool_name": "tool_1",
        "tool_output": "abc.com"
    },
    {
        "project": "a",
        "run": 1,
        "tool_name": "tool_1",
        "tool_output": "xyz.com"
    },
    {
        "project": "a",
        "run": 2,
        "tool_name": "tool_1",
        "tool_output": "abc.com"
    },
    {
        "project": "a",
        "run": 2,
        "tool_name": "tool_1",
        "tool_output": "xyz.com"
    },
    {
        "project": "a",
        "run": 2,
        "tool_name": "tool_1",
        "tool_output": "new.com"
    }
]

```

I need to find tool output differences between this 2 runs - new tool outputs appearing in run 2. I want to get result like this:

```
{
    "new_tool_outputs_count": 1,
    "new_tool_outputs": ["new.com"]
}

```

In SQL analogy it is something like outer join.

 ![Screenshot 2020-06-16 at 14.54.15](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fef510f8ca20ef839090256bed92b26914212f89.jpeg)

I probably need to use some aggregation query, but I am not sure is something like this possible with elasticsearch. Also, the number of documents for comparison can reach a scale of millions.

---

<div class="post-metadata">

### Author: ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [June 16, 2020, 1:32pm UTC](https://discuss.elastic.co/t/differences-between-2-groups-of-documents-by-multiple-criteria/237300/2 "2020-06-16T13:32:45Z")

</div>

Top of mind I think you could solve this in 2 stages with [transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html). In the 1st stage you collapse (`group_by`) project/runs, so you have 1 document per project and run.

In the 2nd stage you compare the runs, we have an example in the docs for [comparing indices](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-compare).

Whether you use 1 transform overall or 2 depends on the number of projects and the number of runs you have per project: If its always only 2 runs, I think it can be done with 1 transform. In this case I would only `group_by` project.

I hope this gives you some idea, scale shouldn't be a problem.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 14, 2020, 1:32pm UTC](https://discuss.elastic.co/t/differences-between-2-groups-of-documents-by-multiple-criteria/237300/3 "2020-07-14T13:32:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.