Mutate rename issue after upgrade

Elastic Stack Logstash
1

Hello,

I have a pipeline where that receives data from a F5 load balancer and parses it with the cef codec, in the filters I have a series of mutate using rename to change the name of some fields to their ecs correspondent, for example requestMethod is renamed to http.request.method and response_code is renamed to http.response.status_code.

Today I upgraded from 7.9.3 to 7.12.1 and one of the renames strangely stopped working, which broke some visualizations and alerts.

The renames appears in the pipeline in the following order among some other rename operations:

            other renames
            rename => { "requestMethod" => "[http][request][method]"}
            rename => { "response_code" => "[http][response][status_code]"}
            other renames

After the upgrade the rename for http.response.status_code stopped working, I tried to move it further down inside the mutate filter, but it didn't work, I needed to change the rename to an add_field to bring back the field into my documents as it is needed.

            other renames
            rename => { "requestMethod" => "[http][request][method]"}
            add_field => { "[http][response][status_code]" => "%{response_code}" }
            other renames

I tried to replicate the issue in my lab, but the problem did not occur.

Has anyone seen anything like this before or have some tips of what I should investigate?

I use a lot of renames in my pipelines and now I will need to check everyone of them for this kind of issue.

2

Does it make any difference if you do

        rename => {
            "requestMethod" => "[http][request][method]"
            "response_code" => "[http][response][status_code]"
        }

I never use multiple instances of an option on a filter because very occasionally logstash combines them in unexpected ways.

3

I think you are right, something weird is happening when logstash is combining those multiple rename options.

I have a big mutate block with multiple rename options, I've always used rename like this and never reached any weird issue until now.

I tried to change the order and the same field was still missing, tried to rename both fields in the same rename option and the same thing happened.

What solved the issue was to use a single rename option inside a new mutate block.

mutate {
        rename => {
            "requestMethod" => "[http][request][method]"
            "response_code" => "[http][response][status_code]"
        }
}

I will review all my pipelines to correct this, what is weird is that I wasn't able to replicate it yet, probably it is a issue that will only happen on some specific conditions like the number of renames for example.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.