Copying field in from one event to other in logstash

Elastic Stack Logstash
1

Can anyone tell me if I can copy a field from one event to another in logstash based on a unique_key present in both the events?

2

You might be able to do it with an aggregate filter.

3

can you please share the config... I have been trying to solve this from a long time now..

4

That depends on what the events look like.

5

Should I share the logs?

6

Yes.

7

These are the logs:
{
{
"_index": "agent",
"_type": "doc",
"_id": "GYmqSGkB5QCX5-g3yFpy",
"_score": 1,
"_source": {
"host": "C02X5KQ4JG5H.group.on",
"@version": "1",
"@timestamp": "2019-03-04T12:25:45.734Z",
"_time": "2019-02-20T09:02:16.842+0000”,
"Email": "c_adimlich@xyz.com",
"Status": “break,
"preview": false,
"FirstName": "Adil "
},
{
"_index": "agent",
"_type": "doc",
"_id": "24mqSGkB5QCX5-g3yFYF",
"_score": 1,
"_source": {
"host": "C02X5KQ4JG5H.group.on",
"@version": "1",
"@timestamp": "2019-03-04T12:25:45.588Z",
"_time": "2019-02-20T09:02:14.842+0000”,
"Email": "c_hesbahi@xyz.com",
"Status": "Unavailable",
"preview": false,
"FirstName": “Hamza”
},
{
"_index": "agent",
"_type": "doc",
"_id": "GYmqSGkB5QCX5-g3yFpy",
"_score": 1,
"_source": {
"host": "C02X5KQ4JG5H.xyz.on",
"@version": "1",
"@timestamp": "2019-03-04T12:25:45.734Z",
"_time": "2019-02-20T09:02:18.766+0000”,
"Email": "c_adimlich@xyz.com",
"Status": “eating,
"preview": false,
"FirstName": "Adil"
}
}

Is there a way in which we can merge the logs with the Firstname as "Adil" ? Here email can work as a unique key.

8

If you use

    aggregate {
        task_id => "%{Email}"
        code => '
            if map["FirstName"]
                event.set("FirstName", map["FirstName"])
            else
                map["FirstName"] = event.get("FirstName")
            end
        '
    }

then every event for each Email will have the same FirstName. However, since order is not preserved you cannot be sure which FirstName the filter will see first.

Note that you must have --pipeline.workers set to 1.

9

Thanks for the solution Badger.
I tried this earlier but this is creating some extra logs with multiple values in a field and that too similar values something like:

36%20PM
Screen Shot 2019-03-04 at 7.53.36 PM.png1010×728 77.4 KB

And yes you are right, order is not preserved maybe that's why its showing this.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.