Correct way to use the mutate rename filter

Elastic Stack Logstash
1

Hello, this is weird,

For the last couple of years I've been using the mutate rename filter in the following way, with one rename option for every field inside the same mutate block.

mutate {
    rename => { "field1" => "newField1" }
    rename => { "field2" => "newField2" }
    rename => { "fieldN" => "newFieldN" }
}

I've never had any problem until yesterday when I upgraded logstash to version 7.12.1 and it broke some of my pipelines because the rename stopped working for some fields.

Following the suggestion on this post, I've changed the rename filter to have multiple fields inside the same mutate block.

I did this for the field that were not been renamed, which was [http][response][status_code].

mutate {
        rename => {
            "requestMethod" => "[http][request][method]"
            "response_code" => "[http][response][status_code]"
        }
}

This worked at the time, today I decided to refactor my pipeline to have a single mutate block with a single rename option with all the fields that I need to rename and then some fields stopped being renamed again.

This was the mutate/rename block that is not working.

mutate {
    rename => { 
        "deviceCustomString5" => "x_forwarded_for_header_value" 
        "deviceCustomString4" => "attack_type"
        "deviceCustomString3" => "full_request"
        "deviceCustomString2" => "http_class_name"
        "deviceCustomString1" => "policy_name"
        "deviceCustomNumber3" => "device_id"
        "deviceCustomNumber2" => "violation_rating"
        "deviceCustomNumber1" => "response_code"
        "deviceCustomDate1" => "policy_apply_date"
        "deviceCustomIPv6Address4" => "ip_address_intelligence"
        "deviceCustomIPv6Address3" => "[destination][ipv6][ip]"
        "deviceCustomIPv6Address2" => "[source][ipv6][ip]"
        "deviceCustomIPv6Address1" => "[device][ipv6][ip]"
        "deviceAddress" => "[device][ip]"
        "deviceAction" => "[event][action]"
        "attack_type" => "[event][type]"
        "deviceEventClassId" => "[event][category]"
        "applicationProtocol" => "[network][protocol]"
        "requestUrl" => "[url][path]"
        "destinationAddress" => "[destination][ip]"
        "destinationPort" => "[destination][port]"
        "destinationUserId" => "[user][id]"
        "destinationUserName" => "[user][name]"
        "sourceAddress" => "[source][ip]"
        "sourcePort" => "[source][port]"
        "sourceUserId" => "[user][id]"
        "sourceUserName" => "[user][name]"
        "requestMethod" => "[http][request][method]"
        "response_code" => "[http][response][status_code]"
    }
}

Some fields, more specifically [http][response][status_code and [event][type] were not renamed.

To make it work I needed to use other mutate blocks to rename the event and http fields.

mutate {
    rename => {
        "deviceAction" => "[event][action]"
        "attack_type" => "[event][type]"
        "deviceEventClassId" => "[event][category]"
    }
}
mutate {
    rename => {
        "requestMethod" => "[http][request][method]"
        "response_code" => "[http][response][status_code]"
    }
}

My question is, is this a bug in version 7.12? I see no reason for the first configuration to not work. What is the correct way to use the rename filter?

Since I use a lot of renames in my pipelines I will need to review each one of them to see if something broke after the upgrade.

What I'm thinking to do is to use a mutate block for every group of fields or nested fields, to help isolate them from wherever is happening with the mutate rename filter.

2

I think from 7.12 it has change to this. your original field is also needs to be inside
I have following and it is working.
I am running 7.12.0

 mutate { rename => {"[type]" => "[system][gpu][type]"  }
               rename => {"[gpuindex]" => "[system][gpu][gpuindex]"  }
               rename => {"[vendor]" => "[system][gpu][vendor]"  }
               rename => {"[gpu_bus_id]" => "[system][gpu][gpu_bus_id]"  }
               rename => {"[series]" => "[system][gpu][series]"  }
               rename => {"[sku]" => "[system][gpu][sku]"  }
               rename => {"[utilization]" => "[system][gpu][utilization]"  }
               rename => {"[temperature][junction]" => "[system][gpu][temperature]"  }
               rename => {"[beat][version]" => "[system][gpu][beat][version]"  }
               rename => {"[count]" => "[system][gpu][count]"  }
               rename => {"[memory]" => "[system][gpu][memory]"  }
               rename => {"[power]" => "[system][gpu][power]"  }
               rename => {"[clocks][pcie]" => "[system][gpu][clocks][pcie]"  }
}
3

I will test that later, but it makes no sense for some of them to work and others do not work.

I have a lot of mutates rename that are still working the same way as before, just a couple of them stopped working and I wasn't able to replicate on my lab yet.

Will try later to create a large sample message with a lot of fields and see if I can catch anything weird.

4

I just noticed this! You are assuming the first listed rename happens before the second. I can definitely believe the validity of that assumption changed. You would need to use two completely separate mutate filter to make sure of the order.

1 Like
5

Oh, I spent all day looking at those pipelines and did not saw this :man_facepalming:

Yeah, my mistake, there is no weird issue after all, I just forgot that the order is not guaranteed within the same option.

Will fix those redundant renames.

Thanks for the catch!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.