Copying field in from one event to other in logstash

[Pranav (Pranav Sharma)]

Can anyone tell me if I can copy a field from one event to another in logstash based on a unique_key present in both the events?

[Badger]

You might be able to do it with an aggregate filter.

[Pranav (Pranav Sharma)]

can you please share the config... I have been trying to solve this from a long time now..

[Badger]

That depends on what the events look like.

[Pranav (Pranav Sharma)]

Should I share the logs?

[Badger]

Yes.

[Pranav (Pranav Sharma)]

These are the logs:
{
{
"_index": "agent",
"_type": "doc",
"_id": "GYmqSGkB5QCX5-g3yFpy",
"_score": 1,
"_source": {
"host": "C02X5KQ4JG5H.group.on",
"@version": "1",
"@timestamp": "2019-03-04T12:25:45.734Z",
"_time": "2019-02-20T09:02:16.842+0000”,
"Email": "c_adimlich@xyz.com",
"Status": “break,
"preview": false,
"FirstName": "Adil "
},
{
"_index": "agent",
"_type": "doc",
"_id": "24mqSGkB5QCX5-g3yFYF",
"_score": 1,
"_source": {
"host": "C02X5KQ4JG5H.group.on",
"@version": "1",
"@timestamp": "2019-03-04T12:25:45.588Z",
"_time": "2019-02-20T09:02:14.842+0000”,
"Email": "c_hesbahi@xyz.com",
"Status": "Unavailable",
"preview": false,
"FirstName": “Hamza”
},
{
"_index": "agent",
"_type": "doc",
"_id": "GYmqSGkB5QCX5-g3yFpy",
"_score": 1,
"_source": {
"host": "C02X5KQ4JG5H.xyz.on",
"@version": "1",
"@timestamp": "2019-03-04T12:25:45.734Z",
"_time": "2019-02-20T09:02:18.766+0000”,
"Email": "c_adimlich@xyz.com",
"Status": “eating,
"preview": false,
"FirstName": "Adil"
}
}

Is there a way in which we can merge the logs with the Firstname as "Adil" ? Here email can work as a unique key.

[Badger]

If you use

    aggregate {
        task_id => "%{Email}"
        code => '
            if map["FirstName"]
                event.set("FirstName", map["FirstName"])
            else
                map["FirstName"] = event.get("FirstName")
            end
        '
    }

then every event for each Email will have the same FirstName. However, since order is not preserved you cannot be sure which FirstName the filter will see first.

Note that you must have --pipeline.workers set to 1.

[Pranav (Pranav Sharma)]

Thanks for the solution Badger.
I tried this earlier but this is creating some extra logs with multiple values in a field and that too similar values something like:

And yes you are right, order is not preserved maybe that's why its showing this.