Mutate filter - Convert a hash data-type to something else

[luiz.miguelsl (Luiz Lima)]

Hey there guys, How are you doing?

I came across this problem and thought you'd be able to help me figure it out. I created a CSV filter to parse my httpd access log file and it's working fine. We have a field "HttpStatus" in our csv filter and this field is sent to ELK as a string. We're trying to create a copy of this field using addfield, keeping the original one unchanged and convert this new field to integer. I saw in the logstash documentation that the data type we get out from a addfield operation is hash, and that the convert operation cant't perform anything when the datatype is hash. How can we acomplish this?

https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-convert

ENV: Logstash 6.5.0

config file:
file {
path => "/path/to/logs/something.access_log*"
start_position => beginning
exclude => "*.gz"
}
}

filter {
    csv {
        columns => [
          "CIP",
          "RemoteLog",
          "RemoteUser",
          "Time",
          "CSUristem",
          "HttpStatus",
          "ResponseSize",
          "CSReferer",
          "UserAgenst",
          "CsBytes",
          "ScBytes",
          "timetakenSegundos",
          "timetakenMicroSegundos"
        ]
        separator => "|"
        }

#### DataType convertion

  mutate {
    convert => { "timetakenSegundos" => "integer" }
    convert => { "timetakenMicroSegundos" => "integer" }
    convert => { "CsBytes" => "integer" }
    convert => { "ScBytes" => "integer" }
    add_field => {http_int => "%{HttpStatus}"}
    convert => { "http_int" => "integer" }

[Jenni]

Where did you read that the output of add_field is a hash? It's a normal string. Your input for that operation is a hash because you have to give it a key and a value for every field that has to be added.

I think your actual problem is more likely the fact that you try to add and convert the field in the same filter. But the execution order of mutate is not the order of your parameters. In fact there is no guaranteed order, so you have to split up operations that rely on each other. The correct syntax would be

mutate {
  convert => { 
    "timetakenSegundos" => "integer"
    "timetakenMicroSegundos" => "integer"
    "CsBytes" => "integer"
    "ScBytes" => "integer"
  }
  add_field => { http_int => "%{HttpStatus}" }
}
mutate {
  convert => { "http_int" => "integer" }
}

But to go even further: add_field one of the default options for every kind of filter and is only executed if the filter has been successful, so you might want to use copy instead.

[Badger]

@Jenni

there is no guaranteed order

The order is set in the code. For the mutate specific options that is here. For the common options it is here.

[Jenni]

Yeah, fair enough. The code does have to have an order. I just meant that I wouldn't rely on it to stay the same as long as the official documentation doesn't specify this But in principle you are right.

Edit: But I haven't read the documentation in a while. It does say it by now! (And copy is after convert and add_field comes last anyway)
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-proc_order

[luiz.miguelsl (Luiz Lima)]

Thanks, Jenni. I misunderstood the hash part of the documentation, but it's clear now. As soon as I did what you suggested, it worked.

You're the best