# Merging documents based on matched fields values

**URL:** <https://discuss.elastic.co/t/merging-documents-based-on-matched-fields-values/249728>\
**Category:** Elasticsearch\
**Created:** [September 24, 2020, 2:47am UTC](https://discuss.elastic.co/t/merging-documents-based-on-matched-fields-values/249728 "2020-09-24T02:47:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![abregman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abregman/32/63862_2.png) [@abregman](https://discuss.elastic.co/u/abregman)\
**Post date:** [September 24, 2020, 2:47am UTC](https://discuss.elastic.co/t/merging-documents-based-on-matched-fields-values/249728/1 "2020-09-24T02:47:47Z")

</div>

I'll start by saying that I'm not sure what I need is actually to merge documents, it's simply the most suitable subject I've found.

Every document created in my environment today has the following fields: job\_name and build\_number so documents look like this:

```auto
{ 'job_name': x,
  'build_numer': 1
  'extra_field_one': ...
}

{ 'job_name': x,
  'build_numer': 1
  'extra_field_two': ...
}

{ 'job_name': x,
  'build_numer': 2
  'extra_field_one': ...
}

```

Now, I would like to treat all the documents with the same job\_name and build\_number as the same document. So basically when I create visualization or run a query, I would like it to run on this document (although it doesn't exists):

```auto
{ 'job_name': x,
  'build_numer': 1
  'extra_field_one': ...
  'extra_field_two': ...
}

```

How can I achieve that? I've seen topics like "top hit aggregations", "collapsed fields", ... but at this point I'm quite confused as to what is the right approach so eventually any user that would run a query or create a visualization, will not have to think about it but the system will be configured to treat documents with the same fields values as one document.

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [September 24, 2020, 6:24am UTC](https://discuss.elastic.co/t/merging-documents-based-on-matched-fields-values/249728/2 "2020-09-24T06:24:17Z")

</div>

This can be achieved by grouping docs using `job_name` and `build_number` and than aggregate the docs with a `scripted_metric`. A perfect fit for that is a [composite aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html). `job_name` and `build_number` are the sources, the `scripted_metric` that merges the fields your aggregation.

However, you want to visualize on top of that, so I think you want to have the merged documents in an index. That's where [transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) comes into play. It basically runs a composite aggregation, takes the results and writes the output into a new index. You can set it up to run continuously, so it will consume your new incoming data (you need a timestamp for that, but I guess you have that). In transform `job_name` and `build_number` would be your `group_by`.

Regarding the `scripted_metric` aggregation, please have a look [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html). The examples are not transform specific, but work in aggregations.

Here is a more specific example, I found:

```auto
"scripted_metric": {
          "init_script": "state.join = new HashMap()",
          "map_script": "String[] fields = new String[] {'name', 'best_friend', 'hobby'}; for (e in fields) { if (doc.containsKey(e)) {state.join.put(e, doc[e])}}",
          "combine_script": "return state.join",
          "reduce_script": "String[] fields = new String[] {'name', 'best_friend', 'hobby'}; Map j=new HashMap(); for (s in states) {for (e in fields) { if (s.containsKey(e)) {j.put(e, s[e].get(0))}}} return j;"
        }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2020, 6:24am UTC](https://discuss.elastic.co/t/merging-documents-based-on-matched-fields-values/249728/3 "2020-10-22T06:24:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.