# Merging fields of documents based on some field

**URL:** <https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413>\
**Category:** Logstash\
**Created:** [September 27, 2019, 2:00pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413 "2019-09-27T14:00:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [September 27, 2019, 2:00pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/1 "2019-09-27T14:00:39Z")

</div>

Hi Elasticians,  
is it possible using Logstash to do this:

```
#fields of document 1
connection-id: 1000
ip: 10.88.88.201

#fileds of document 2
connection-id: 1000
uid: user

#fields of document 3
connection-id: 1000
message: Login Failed

```

I would like to create document from these 3 documents. Result document should look like this:

```
#resulted field
connection-id: 1000
ip: 10.88.88.201
message: Login Failed
uid: user

```

Merging fields should be based on `connection-id` field. Is it possible to do with [Logstash - Aggregate Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-code)?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 27, 2019, 2:24pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/2 "2019-09-27T14:24:30Z")

</div>

> [@vasek](#):
>
> Is it possible to do with [Logstash - Aggregate Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-code)?

Yes. You probably want to model your code after Example 3 on that page.

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [September 27, 2019, 2:54pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/3 "2019-09-27T14:54:33Z")

</div>

Example #3 looks good. I have no idea what to put to the `code` section to achieve merging of fields.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 27, 2019, 7:58pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/4 "2019-09-27T19:58:41Z")

</div>

If you are using push\_map\_as\_event\_on\_timeout then you need the map to contain the fields that you want in the final event

```
code => '
    ip = event.get("ip")
    if ip
        map["ip"] = ip
    end
    # and similary for the other fields

    event.cancel # Assuming you only want the aggregated events
'
```

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [September 28, 2019, 1:45pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/5 "2019-09-28T13:45:40Z")

</div>

Thank you Badger. It works like a charm 😉

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [September 28, 2019, 2:41pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/6 "2019-09-28T14:41:14Z")

</div>

Hi @Badger,  
I am trying to create nested json field user.field but I cannot find syntax for map.

#this is not nested json

```
map['user.list']

```

#this create only list field

```
map['[user][list]']

```

#this create nested json field but this field is not present in aggregated event - only on events that is part of aggregation

```
event.set('[foo][bar][c]', [3, 4])
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 28, 2019, 3:24pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/7 "2019-09-28T15:24:41Z")

</div>

When you say you want a nested field you are saying that you want the map entry to be a hash. You can do that using

```
code => 'map["user"] = { "list" => "foo" }'
```

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [September 29, 2019, 4:58am UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/8 "2019-09-29T04:58:33Z")

</div>

> [@Badger](#):
>
> code =\> 'map["user"] = { "list" =\> "foo" }'

Thank @Badger for you reply. The code above produced this:

```
"user": [
  {
    "list": "admin"
  },
  {
    "list": "root"
  }
],

```

but I would like to achieve this:

```
  "user": {
    "list": [
      "admin",
      "root"
    ]
  }

```

Do you have any idea how to do it?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 29, 2019, 12:52pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/9 "2019-09-29T12:52:19Z")

</div>

```
code => '
    map["user"] ||= { "list" => [] }
    map["user"]["list"] << "foo"
    map["user"]["list"] << "bar"
'
```

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [September 29, 2019, 1:28pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/10 "2019-09-29T13:28:44Z")

</div>

> [@Badger](#):
>
> code =\> ' map["user"] ||= { "list" =\> } map["user"]["list"] \<\< "foo" map["user"]["list"] \<\< "bar" '

Thank you so much @Badger. It works fine. You saved me a lot of time! 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2019, 1:28pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413/11 "2019-10-27T13:28:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.