Trouble Mutating

Elastic Stack Logstash
1

I am having trouble performing a mutate. I thought I had it working with a more simplified version of this config but cannot seem to get it going in this new one (though everything else seems to be working much better).

    convert => {
            "Meter" => "integer"
            "Group" => "integer"
            "Org" => "integer"
            "Reading" => "integer"
            "Info1" => "integer"
            "Info2" => "integer"
            "Info3" => "integer"
            "Info4" => "integer"
            "Info5" => "integer"
            "Info6" => "integer"
            "Info7" => "integer"
            }

This produces an error as follows:

[WARN ] 2020-11-16 16:02:03.348 [[main]>worker1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"test-west", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x74d535a], :response=>{"index"=>{"_index"=>"test-west-coop2", "_type"=>"_doc", "_id"=>"PCMU03UBcHG8wQHZR9GN", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [Meter] cannot be changed from type [long] to [text]"

But when I add mutate either encapsulating the converts or when I separate out the mutate into its own filter function, I receive other errors.

    mutate {
            convert => { "Reading" => "integer" }
    convert => {
            "Meter" => "integer"
            "Group" => "integer"
            "Org" => "integer"
            "Info1" => "integer"
            "Info2" => "integer"
            "Info3" => "integer"
            "Info4" => "integer"
            "Info5" => "integer"
            "Info6" => "integer"
            "Info7" => "integer"
            }

            remove_field => [
            "message",
            "host",
            "path",
            "@version"
                    ]

    }
    }

or

    mutate {
    convert => {
            "Meter" => "integer"
            "Group" => "integer"
            "Org" => "integer"
            "Reading" => "integer"
            "Info1" => "integer"
            "Info2" => "integer"
            "Info3" => "integer"
            "Info4" => "integer"
            "Info5" => "integer"
            "Info6" => "integer"
            "Info7" => "integer"
            }

            remove_field => [
            "message",
            "host",
            "path",
            "@version"
                    ]

    }
    }

I receive the following:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "=>" at line 26, column 9 (byte 530) after filter {\n csv {\n columns => [\n "Meter",\n "Group",\n "Org",\n "date",\n "Reading",\n "Info1",\n "Info2",\n "Info3",\n "Info4",\n "Info5",\n "Info6",\n "Info7"\n ]\n\n\tmutate ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in block in converge_state'"]}

I'm pretty new to all this so I'm probably just doing something wrong but any pointers in the right direction would be greatly appreciated.

2

For this one you are missing a } to close the csv filter before starting the mutate filter.

"error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [Meter] cannot be changed from type [long] to [text]"

Sounds like you are relying on elasticsearch to do dynamic type detection. You may want to add an index template that sets the type of the field.

The error message comes from elasticsearch, so if you want to gain a better understanding of exactly what that message is telling you then you may need to ask in the elasticsearch forum.

3

Thank you @Badger. The trouble is I cannot seem to find where I'm not terminating with }

I think it's with my syntax or where i'm putting the remove_field. Below is an updated config that works:

filter {
csv {
columns => [
"Meter",
"Group",
"Org",
"date",
"Reading",
"Info1",
"Info2",
"Info3",
"Info4",
"Info5",
"Info6",
"Info7"
]
separator => ","
}
date {
"target" => "DateofReading"
"match" => ["date" , "yyyy-M-d H"]
"timezone" => "America/Chicago"
}

    mutate {
            convert => { "Meter" => "integer" }
            convert => { "Group" => "integer" }
            convert => { "Org" => "integer" }
            convert => { "Reading" => "integer" }
            convert => { "Info1" => "integer" }
            convert => { "Info2" => "integer" }
            convert => { "Info3" => "integer" }
            convert => { "Info4" => "integer" }
            convert => { "Info5" => "integer" }
            convert => { "Info6" => "integer" }
            convert => { "Info7" => "integer" }
            }

}

But adding the remove_field as in below:

filter {
csv {
columns => [
"Meter",
"Group",
"Org",
"date",
"Reading",
"Info1",
"Info2",
"Info3",
"Info4",
"Info5",
"Info6",
"Info7"
]
separator => ","
}
date {
"target" => "DateofReading"
"match" => ["date" , "yyyy-M-d H"]
"timezone" => "America/Chicago"
}

    remove_field => {
  	"message",
  	"host",
  	"path",
  	"@version"
  			}

    mutate {
            convert => { "Meter" => "integer" }
            convert => { "Group" => "integer" }
            convert => { "Org" => "integer" }
            convert => { "Reading" => "integer" }
            convert => { "Info1" => "integer" }
            convert => { "Info2" => "integer" }
            convert => { "Info3" => "integer" }
            convert => { "Info4" => "integer" }
            convert => { "Info5" => "integer" }
            convert => { "Info6" => "integer" }
            convert => { "Info7" => "integer" }
            }

}

results in the following error:

agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "{" at line 33, column 22 (byte 706) after filter {\n csv {\n columns => [\n "Meter",\n "Group",\n "Org",\n "date",\n "Reading",\n "Info1",\n "Info2",\n "Info3",\n "Info4",\n "Info5",\n "Info6",\n "Info7"\n ]\n separator => ","\n }\n date {\n "target" => "DateofReading"\n "match" => ["date" , "yyyy-M-d H"]\n "timezone" => "America/Chicago"\n }\n\n remove_field ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in block in converge_state'"]}

I suppose this is less of a problem with mutating and more of a lack of knowledge on my part in how to structure these config files. I've been through several examples and tried so many permutations that I've lost count. Does the remove_field need to be within the csv attribute or maybe the remove_field section itself contained in brackets?

4

oh, and I am taking your suggestion to use templates...however in my brief dive into them this morning, I realize that they're a bit over my head at the moment. Ultimately I'll want to do that but for the time being, I just want to get this in there for a one-time import/proof-of-concept.

5

I don't get why, but for some reason putting the remove_field section within the date attribute brackets seemed to work. I tried putting it in the mutate brackets and even its own filter brackets but neither of those worked...however putting it within the date brackets seemed to do the trick. I'd like to know why...but I guess it's no big deal as long as it's working.

filter {
csv {
columns => [
"Meter",
"Group",
"Org",
"date",
"Reading",
"Info1",
"Info2",
"Info3",
"Info4",
"Info5",
"Info6",
"Info7"
]
separator => ","
}
date {
"target" => "DateofReading"
"match" => ["date" , "yyyy-M-d H"]
"timezone" => "America/Chicago"
remove_field => [ "message","host","path","@version" ]
}
mutate {
convert => { "Meter" => "integer" }
convert => { "Group" => "integer" }
convert => { "Org" => "integer" }
convert => { "Reading" => "integer" }
convert => { "Info1" => "integer" }
convert => { "Info2" => "integer" }
convert => { "Info3" => "integer" }
convert => { "Info4" => "integer" }
convert => { "Info5" => "integer" }
convert => { "Info6" => "integer" }
convert => { "Info7" => "integer" }
}

}

6

remove_field is not a filter by itself, it is an option that all filters support, so it does need to be inside a filter.

1 Like
7

Thank you for your help @Badger. Been awhile since I felt like such a noob but this stuff is a bit different from my typical role as a SysAdmin. I'm just trying to show my coworkers that there's more efficient ways to deal with data than breaking it up into chunks that excel can work with. Thanks again for your help and patience.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.