Merging fields of documents based on some field

Hi Elasticians,
is it possible using Logstash to do this:

#fields of document 1
connection-id: 1000
ip: 10.88.88.201

#fileds of document 2
connection-id: 1000
uid: user

#fields of document 3
connection-id: 1000
message: Login Failed

I would like to create document from these 3 documents. Result document should look like this:

#resulted field
connection-id: 1000
ip: 10.88.88.201
message: Login Failed
uid: user

Merging fields should be based on connection-id field. Is it possible to do with Logstash - Aggregate Plugin?

Yes. You probably want to model your code after Example 3 on that page.

1 Like

Example #3 looks good. I have no idea what to put to the code section to achieve merging of fields.

If you are using push_map_as_event_on_timeout then you need the map to contain the fields that you want in the final event

code => '
    ip = event.get("ip")
    if ip
        map["ip"] = ip
    end
    # and similary for the other fields

    event.cancel # Assuming you only want the aggregated events
'
1 Like

Thank you Badger. It works like a charm :wink:

Hi @Badger,
I am trying to create nested json field user.field but I cannot find syntax for map.

#this is not nested json

map['user.list']

#this create only list field

map['[user][list]']

#this create nested json field but this field is not present in aggregated event - only on events that is part of aggregation

event.set('[foo][bar][c]', [3, 4])

When you say you want a nested field you are saying that you want the map entry to be a hash. You can do that using

code => 'map["user"] = { "list" => "foo" }'

Thank @Badger for you reply. The code above produced this:

"user": [
  {
    "list": "admin"
  },
  {
    "list": "root"
  }
],

but I would like to achieve this:

  "user": {
    "list": [
      "admin",
      "root"
    ]
  }

Do you have any idea how to do it?

code => '
    map["user"] ||= { "list" => [] }
    map["user"]["list"] << "foo"
    map["user"]["list"] << "bar"
'
1 Like

Thank you so much @Badger. It works fine. You saved me a lot of time! :slightly_smiling_face:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.