# \[SOLVED\] The filter plugin mutate-convert doesn't work in 5.0

**URL:** <https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496>\
**Category:** Logstash\
**Created:** [November 9, 2016, 1:28pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496 "2016-11-09T13:28:51Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 9, 2016, 1:28pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/1 "2016-11-09T13:28:51Z")

</div>

In my logstash config I have something like this:

```
        mutate {
            add_field => { "[%{name}][long]" => "%{valueLong}" }
            remove_field => [valueLong]
            convert => ["[%{name}][long]", "integer"]
        }

```

And the new field should be an integer. Everything works as expected, the new field is added, the old field deleted, but the new field is always text.

This configuration worked without problems before the new Logstash version 5.0. We want to update to the newer version, but we get stuck on this configuration problem. Does anyone has an idea how to solve this?

Thank you.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 9, 2016, 2:18pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/2 "2016-11-09T14:18:41Z")

</div>

You can't assume that `add_field`, `remove_field`, and `convert` will execute in the order given. You have to split your mutate filter in three.

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 9, 2016, 2:39pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/3 "2016-11-09T14:39:50Z")

</div>

Still doesn't work. I try to explain the problem again:  
I have a log file with key-value entries. Something like:

timestamp="18.07.2016 15:48:09,614" id="3" name="duration" type="DURATION" valueType="long" valueLong=79

Using logstash I write in two indices in Elastic:

- one that contains the row data, something like:

- and the second one, we call it summary index and the documents inside look like:

If another entry with the same ID will be logged, the first index will contain another document, but the second will update the existing one. Normally when this happens, the name is different and another structure will be created.

And exactly this functionality worked with Logstash \< 5.0 using this code:

```
   mutate {
        add_field => { "[%{name}][long]" => "%{valueLong}" }
        remove_field => [valueLong]
        convert => ["[%{name}][long]", "integer"]
    }

```

I changed now, as you proposed to ...

```
        mutate {
            add_field => { "[%{name}][long]" => "%{valueLong}" }
        }
        mutate {
            remove_field => [valueLong]
        }
        mutate {
            convert => ["[%{name}][long]", "integer"]
        }

```

... but no change at all. duration.long is still a text.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 9, 2016, 2:55pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/4 "2016-11-09T14:55:27Z")

</div>

`valueLong` already is an integer field so I don't know why you're doing the conversion in the first place. When you say

> ... but no change at all. duration.long is still a text.

are you saying that the field in Elasticsearch (and, by extension, Kibana) is a string field? In that case you have to reindex the current index or create a new index. The existing mapping won't change just because you're starting to submit documents with an integer `[duration][long]` field.

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 9, 2016, 3:05pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/5 "2016-11-09T15:05:06Z")

</div>

In my example I wanted to show you, what I actually expect in elastic.

I delete all indexes, before every test.

I tested now without the conversion, but no change. Even if the `valueLong` field is an integer, the `duration.long` field is a string in the second index.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 9, 2016, 3:07pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/6 "2016-11-09T15:07:42Z")

</div>

What do the mappings look like? What index templates do you have?

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 9, 2016, 3:17pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/7 "2016-11-09T15:17:10Z")

</div>

The mappings look like: (I deleted some other attributes that are not importante for the discussion). The mappings are dynamically generated.

> {  
> "logstash-summary" : {  
> "mappings" : {  
> "SUMMARY" : {  
> "\_all" : {  
> "enabled" : true,  
> "norms" : false  
> },  
> "dynamic\_templates" : [  
> {  
> "message\_field" : {  
> "path\_match" : "message",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "norms" : false,  
> "type" : "text"  
> }  
> }  
> },  
> {  
> "string\_fields" : {  
> "match" : "_",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> },  
> "norms" : false,  
> "type" : "text"  
> }  
> }  
> }  
> ],  
> "properties" : {  
> "@timestamp" : {  
> "type" : "date",  
> "include\_in\_all" : false  
> },  
> "@version" : {  
> "type" : "keyword",  
> "include\_in\_all" : false  
> },  
> "duration" : {  
> "properties" : {  
> "id" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> },  
> "long" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> },  
> "valueType" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> }  
> }  
> },  
> }  
> }  
> }  
> },  
> "logstash-guardean" : {  
> "mappings" : {  
> "DURATION" : {  
> "\_all" : {  
> "enabled" : true,  
> "norms" : false  
> },  
> "dynamic\_templates" : [  
> {  
> "message\_field" : {  
> "path\_match" : "message",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "norms" : false,  
> "type" : "text"  
> }  
> }  
> },  
> {  
> "string\_fields" : {  
> "match" : "_",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> },  
> "norms" : false,  
> "type" : "text"  
> }  
> }  
> }  
> ],  
> "properties" : {  
> "@timestamp" : {  
> "type" : "date",  
> "include\_in\_all" : false  
> },  
> "@version" : {  
> "type" : "keyword",  
> "include\_in\_all" : false  
> },  
> "id" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> },  
> "name" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> },  
> "type" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> },  
> "valueLong" : {  
> "type" : "long"  
> },  
> "valueType" : {  
> "type" : "text",  
> "norms" : false,  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword"  
> }  
> }  
> },  
> }  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 9, 2016, 3:23pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/8 "2016-11-09T15:23:44Z")

</div>

I still think you have an index template that applies to your new indexes every time. As far as I know Elasticsearch won't default to any dynamic templates so the mappings you have with dynamic templates must come from somewhere.

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 9, 2016, 3:32pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/9 "2016-11-09T15:32:48Z")

</div>

I do nothing in elasticsearch. I suppose that this is the job of logstash. I just have a log file and the logstash with the configuration that I showed you before. What is strange, that I had no problems before 5.0. So I suppose that something have been changed, but I don't know where.

What is also strange is, that I do a mutate-convert before, for the `valueLong` and it works.

I copy again my entire configuration:

```
input {
    file {
        path => "C:/Analytics/analytics.log"
        start_position => "beginning"
    }
}

filter {
    kv { }

    mutate {
        convert => ["valueLong", "integer"]
        convert => ["valueDouble", "float"]
        convert => ["valueBoolean", "boolean"]
    }

    clone {
        clones => ["SUMMARY"]
    }

    if [type] == "SUMMARY" {
        if [valueLong] {
            mutate {
                add_field => { "[%{name}][long]" => "%{valueLong}" }
                remove_field => [valueLong]
                convert => ["[%{name}][long]", "integer"]
            }
        }
        if [valueDouble] {
            mutate {
                add_field => { "[%{name}][double]" => "%{valueDouble}" }
                remove_field => [valueDouble]
                convert => ["[%{name}][double]", "float"]
            }
        }
        if [valueBoolean] {
            mutate {
                add_field => { "[%{name}][boolean]" => "%{valueBoolean}" }
                remove_field => [valueBoolean]
                convert => ["[%{name}][boolean]", "boolean"]
            }
        }
        mutate {
            add_field => { "[%{name}][id]" => "%{id}" }
            add_field => { "[%{name}][valueType]" => "%{valueType}" }
            remove_field => ["name", "id", "valueType"]
        }
    }
}

output {
    stdout { codec => rubydebug }
    if [type] == "SUMMARY" {
        elasticsearch {
            document_id => "%{executionID}"
            action => "update"
            doc_as_upsert => true
            index => "logstash-summary-%{+YYYY.MM.dd}"
            hosts => localhost
        }
    } else {
        elasticsearch {
            index => "logstash-guardean-%{+YYYY.MM.dd}"
            hosts => localhost
        }
    }
}
```

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 10, 2016, 7:04am UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/10 "2016-11-10T07:04:53Z")

</div>

No other idea? ☹

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 11, 2016, 11:57am UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/11 "2016-11-11T11:57:40Z")

</div>

I simplified the example to be easy to understand. Everyone can test this really quick:

Logstash Config:

```
input {
    file {
        path => "C:/Analytics/analytics.log"
        start_position => "beginning"
    }
}

filter {
    kv { }

    mutate {
        convert => ["test", "integer"]
    }

    clone {
        clones => "TEST_CLONE"
    }

    if [type] == "TEST_CLONE" {
        mutate {
            add_field => { "test_clone" => "%{test}" }
            remove_field => [test]
            convert => ["test_clone", "integer"]
        }
    }
}

output {
    stdout { codec => rubydebug }
    if [type] == "TEST_CLONE" {
        elasticsearch {
            index => "logstash-test-clone-%{+YYYY.MM.dd}"
            hosts => localhost
        }
    } else {
        elasticsearch {
            index => "logstash-test-%{+YYYY.MM.dd}"
            hosts => localhost
        }
    }
}

```

The log file contains just one entry:

`test=111`

and the result looks like:

```
{
     "path" => "C:/Analytics/analytics.log",
    "@timestamp" => 2016-11-11T11:52:01.950Z,
    "test" => 111,
    "@version" => "1",
    "host" => "SV-PC-074",
    "message" => "test=111\r"
}
{
    "path" => "C:/Analytics/analytics.log",
    "@timestamp" => 2016-11-11T11:52:01.950Z,
    "test_clone" => "111",
    "@version" => "1",
    "host" => "SV-PC-074",
    "message" => "test=111\r",
    "type" => "TEST_CLONE"
}

```

As you can see the `test` field is a number, but `test_clone` is a string. I think there is a bug, but I'm open for other ideas.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 11, 2016, 12:05pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/12 "2016-11-11T12:05:22Z")

</div>

As I said earlier in the thread you need to split your mutate filter. `add_field` and `remove` applies after `convert`.

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 11, 2016, 12:12pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/13 "2016-11-11T12:12:45Z")

</div>

It works. Strange that I tested before on my complicated configuration and somehow didn't work. I'll try again. Thank you.

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 11, 2016, 12:38pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/14 "2016-11-11T12:38:08Z")

</div>

Now I found the problem in my configuration but not the solution. I copy here just the important part of the configuration (the rest is exactly the same):

```
  if [type] == "TEST_CLONE" {
        mutate {
            add_field => { "[%{name}][long]" => "%{test}" }
        }
        mutate {
            remove_field => [test]
        }
        mutate {
            convert => ["[%{name}][long]", "integer"]
        }
    }

```

The log file contains now the line:

`name="test_clone" test=111`

And now it doesn't work again ☹ The problem is `%{name}`. With an hard coded field name, it works.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 11, 2016, 12:47pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/15 "2016-11-11T12:47:21Z")

</div>

Yes, this is a limitation. For now you'll have to use a ruby filter to convert dynamically named fields.

> <https://github.com/logstash-plugins/logstash-filter-mutate/issues/57>

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 11, 2016, 1:05pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/16 "2016-11-11T13:05:06Z")

</div>

> [@magnusbaeck](#):
>
> For now you'll have to use a ruby filter to convert dynamically named fields

Do you have an example?

---

<div class="post-metadata">

**Author:** ![tileaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tileaga/32/13063_2.png) [@tileaga](https://discuss.elastic.co/u/tileaga)\
**Post date:** [November 11, 2016, 1:21pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/17 "2016-11-11T13:21:31Z")

</div>

I found it 🙂 If someone want to use the workaround, it looks like:

```
 if [type] == "TEST_CLONE" {
        ruby {
            code => 'event.set("[" + event.get("name") + "][long]", event.get("test"))'
        }
        mutate {
            remove_field => [test]
        }
        mutate {
            convert => ["[%{name}][long]", "integer"]
        }
    }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 9, 2016, 1:21pm UTC](https://discuss.elastic.co/t/solved-the-filter-plugin-mutate-convert-doesnt-work-in-5-0/65496/18 "2016-12-09T13:21:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.