The fliter plugin mutate-convert method only convert the first element of an array

Elastic Stack Logstash
1

I used jdbc input plugin to select a column named available_cities from mysql. It was a string field which formatted like this

"12, 13, 143, 123...."

And the filter was

filter {
    mutate {
        split => { "available_cities" => "," }
        convert => { "available_cities" => "integer" }
    }
}

# I also tried this
filter {
    mutate {
        split => { "available_cities" => "," }
    }
    mutate {
        convert => { "available_cities" => "integer" }
    }
}

The output was elasticsearch. My expectations was an available_cities field with multiple values

"available_cities":  [12, 13, 143, 123....]

However, I only got number 12 in field available_cities.
As I read the doc, I do not understand what the problem is.

convert

  • Value type is hash
  • There is no default value for this setting.
    Convert a field’s value to a different type, like turning a string to an integer. If the field value is an array, all members will be converted. If the field is a hash, no action will be taken.
1 Like
2

OMG, I'm misled by the es-head plugin.
It only showed one element on the web page when the field had multiple values.

1 Like
3

I, too, would like to convert a string to an array of integers, and I set up my mutations exactly as you did, and am seeing it converted to only the first element. I don't understand your follow-up: are you saying that you were just seeing a visual artifact? I'm pretty sure I'm not getting the other values: if I search on the other values, I don't find them.

The data in MySQL:

my_field => "1, 2, 3"

My original, simple filter without conversion:

filter {
    mutate {
        split => { "my_field" => "," }
    }
}

The original output in Elasticsearch:

my_field: ["1", "2", "3"]

Searching for myfield: 3 found the item.

But then when I added conversion:

filter {
    mutate {
        split => { "my_field" => "," }
        convert => { "my_field" => "integer" }
    }
}

my output was:

my_field: 1

and searching for myfield: 3 returned no results.

Thoughts?

6

Thanks @sfentress for getting this re-opened (elastic stop autoclosing these).

As I said in private here is the solution:

You need to have two separate mutate statements, 1st one splits the string an creates an array.
2nd one converts the array to the desired type.

filter {
mutate {
split => { "available_cities" => "," }
}
mutate {
convert => { "available_cities" => "integer" }
}
}

This is the type of detail, that needs to be added to the documentation. Just add a note that a "seperate mutate must be performed AFTER the split operation".

Would save us all a lot of time and sanity...
Chris

1 Like
7

Awesome, thank you!