Replacing empty string with dynamic value from the same field

Hi everyone,

I would be grateful for your help regarding how to replace an empty string in a field witha dynamic value. I've already tried different filters such as replace, copy, etc. but it doesn't work as needed. Here is a log example:

2020-07-05 16:43:30.945 [INFO] Product Name: Table
2020-07-05 16:43:30.947 [INFO] Product Material: Wood
2020-07-05 16:43:30.948 [ERROR] Product Color: White

After using GROK filter, I have the following output:
Timestamp logLevel ProductType
2020-07-05 16:43:30.945 [INFO] Table
2020-07-05 16:43:30.947 [INFO] -
2020-07-05 16:43:30.948 [ERROR] -

I'm trying to replace empty string with the productType in order to be able to filter out all relevant information for a particular product based on its type. productType itself appears only ones in the whole log. Currently it doesn't work possibly because of timestamp and since productType is a dynamic field and might be different each time, so I can't assign a static value.
I would be grateful for any ideas on how to do it maybe using lookup or ruby codec.

Desired output would be:

Timestamp logLevel ProductType
2020-07-05 16:43:30.945 [INFO] Table
2020-07-05 16:43:30.947 [INFO] Table
2020-07-05 16:43:30.948 [ERROR] Table

Then comes the nex log and it works similarly but for another ProductType
2020-07-05 17:31:30.945 [INFO] Chair
2020-07-05 17:31:30.947 [INFO] Chair
2020-07-05 17:32:30.948 [ERROR] Chair

Thank you!

If I've got that right, you want to fill an empty field in the following events with the existing value from a previous event. Then your best bet would probably be to use only one worker thread and an aggregate filter.

When the value exists, you save it in the map. When it doesn't, you take it from the map.

Hi Jenni,

Thanks for your idea and help. Yes, you got it right, I need to use the value from the previous event. Regarding the aggregate option I was hesitating since in each log will be a different value and logs will be ingested continiously. And on the web site there is a comment that "events might be processed out of sequence".
If I understand you correctly, it should look like this:
filter {
if "ProductType" in [message ] {
grok { match => { [message] => "%{TIMESTAMP_ISO8601:timestamp} ... %{WORD:productType}"}
}
if [productType] != "" {
aggregate {
productTypeFinal => "%{productType}"
code => "map['productTypeFinal']=0"
map_action => "create"
}
....
map_action =>"update"
}
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.