Some fields are missing after rename.

Elastic Stack Logstash
1

After logstash-oss is upgraded from 7.6.0 to 7.12.1, some fields are lost after parsing the nested JSON data of Kafka.

Kafka JSON example data: {"timestamp":1702630468791,"region":"cn-north-3","eventId":"QER_INFO","args":{"unNum":2,"totalNum":12,"eventId":"QER_INFO"}}

The parsing configuration in Logstash is as follows:

    else if [eventId] == "QER_INFO" {
        mutate {
            rename => {
                "[args][unNum]" => "[__args][unNum]"
                "[args][totalNum]" => "[__args][totalNum]"
                "__args" => "args"
            }
        }
        mutate {
            convert => { "[args][unNum]" => "integer" }
            convert => { "[args][totalNum]" => "integer" }
        }
    }

Why does args contain only the totalNum field in the ES data that is actually imported to the database? No error is found in Logstash run logs.

2

Hello and welcome,

Please share your entire logstash configuration, it is not possible to try to replicate without knowing how you are parsing your data.

Also, this filter is a little confusing, it is not clear what you want to achieve with it:

        mutate {
            rename => {
                "[args][unNum]" => "[__args][unNum]"
                "[args][totalNum]" => "[__args][totalNum]"
                "__args" => "args"
            }
        }

Please also share the output you are getting for the sample message you shared and what is the expected output.

3

You are making assumptions about the order of entries in a hash. In the distant past, that was valid, since Ruby hashes are ordered. But Java hashes are not, so upon the javafication of the pipeline engine, those assumptions became invalid. It looks like you only want to keep two entries within the [args] field. Split this into two rename filters, one for the two entries, one for [__args].

4

Kafka data is reported by the service party. Assume that the data is unreliable. To avoid unexpected exceptions during parsing, the args field is filtered. Only the unNum and totalNum fields are retained and other fields are discarded. and convert it to the integer type. Received with a new field __args and renamed to args

5

The output of the sample message shared is as follows:

      {
        "_index" : "wsk_access__qer_info_2023.12.15",
        "_type" : "_doc",
        "_id" : "PfZTa4wBxI1zWeQYzPgL",
        "_score" : null,
        "_source" : {
          "args" : {
            "totalNum" : 12
          },
          "eventId" : "QER_INFO",
          "@timestamp" : "2023-12-15T16:54:28.791Z",
          "_eventId" : "qer_info",
          "region" : "cn-north-3",
          "timestamp" : 1702630468791,
          "@version" : "1"
        },
        "sort" : [
          1702630468791
        ]
      }

The expected output is:

      {
        "_index" : "wsk_access__qer_info_2023.12.15",
        "_type" : "_doc",
        "_id" : "PfZTa4wBxI1zWeQYzPgL",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2023-12-15T16:54:28.791Z",
          "args" : {
            "unNum" : 2,
            "totalNum" : 12
          },
          "timestamp" : 1702630468791,
          "eventId" : "QER_INFO",
          "@version" : "1",
          "region" : "cn-north-3",
          "_eventId" : "qer_info"
        },
        "sort" : [
          1702630468791
        ]
      }
6

As mentioned You need to use two different mutates, it won't work this way as there is no guarantee on the order of the operations.

You need something like this:

mutate {
    rename => {
        "[args][unNum]" => "[__args][unNum]"
        "[args][totalNum]" => "[__args][totalNum]"
    }
}
mutate {
    rename => {
        "[__args]" => "[args]"
    }
}
7

One more thing, I find if I change the following writing, it will output as I expect:

    else if [eventId] == "QER_INFO" {
        mutate {
            rename => ["[args][unNum]","[__args][unNum]"]
            rename => ["[args][totalNum]","[__args][totalNum]"]
            rename => ["__args","args"]
        }
        mutate {
            convert => { "[args][unNum]" => "integer" }
            convert => { "[args][totalNum]" => "integer" }
        }
    }
8

As mentioned, I would suggest that you use a complete different mutate block, if you are doing multiple mutates on the same field you should use a different mutate block.

This is mentioned in the documentation.

Each mutation must be in its own code block if the sequence of operations needs to be preserved.

9

Do you mean that this way of writing does not work in logstash-oss version 7.12.1?

10

Right, I think it stopped working in 5.0, but I could be wrong about that version number.

Split the three renames into two mutate filters.

11

Thank you very much, this solved my problem

12

Thank you very much, this solved my problem.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.