[SOLVED] Logstash 5.2.2 - Mutate string value to number


(Chris Caputo) #1

I am having an issue with a logstash 5.2.2 filter. I need to convert a string to an integer and it's failing. The approach I am taking may not be ideal. Here's the problem statement: A JMX MBean value reports a "metric_value_string" as "XX.X secs". I want to be able to use this value to be a number for Kibana aggregation in visualizations.

What I'd tried:

Definitions: Source field is "metric_value_string". A string. Values look like "26.0 secs". Desired destination field is "time_in_seconds". An integer.

I tried creating a scripted field. Simply (painless) converting "metric_value_string" to a number looked like a number in Kibana discovery results, was available for aggregation as a number in visualizations. When the visualization is run, an error occurs. It's a cast exception and it shows the values from "metric_value_string", rather than my converted "time_in_seconds". This is Kibana 5.2.2. IDK if this is a bug, so I tried another approach.

I tried creating and converting a field in the logstash filter.

I have tried both of these approaches:

filter {
    if "TimeSince" in [metric_path] or "Delay" in [metric_path] {
        mutate { add_field => { "time_in_seconds" => "%{metric_value_string}"} }
        mutate { gsub => ["time_in_seconds", ".0 secs", ""] }
        mutate { convert => { "time_in_seconds", "integer" } }          
    }
}

and

filter {
    if "TimeSince" in [metric_path] or "Delay" in [metric_path] {
        ruby {
            code =>
                "event.set('time_in_seconds', event.get('metric_value_string'))"
        }
        mutate { gsub => ["time_in_seconds", ".0 secs", ""] }
        mutate { convert => { "time_in_seconds", "integer" } }          
    }
}

The conditional stuff is OK. When I comment out the

mutate { convert => { "time_in_seconds", "integer" } }

code, it looks as expected in output and does not fail.

I am not sure what's causing the failure. The error say's it's a syntax error, but all is OK if I simply comment out the convert line. As a sanity check, I added more code after the gsub line, to ensure that there were no issues with that code.

Here's the STDOUT from logstash:

C:\Elastic\logstash-5.2.2\bin>cls

C:\Elastic\logstash-5.2.2\bin>logstash -f config/logstash.conf --config.reload.automatic 
JAVA_OPTS was set to [ -Dlog4j.configurationFile=C:\Elastic\logstash-5.2.2\config\log4j2.properties     -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="C:\Elastic\logstash-5.2.2/heapdump.hprof"]. Logstash will trust these options, and not set any defaults that it might usually set
[2017-09-14T14:14:14,056][ERROR][logstash.agent           ] Cannot load an invalid configuration {:reason=>"Expected one of #, => at line 14, column 42 (byte 355) after filter {\n\tif \"TimeSince\" in [metric_path] or \"Delay\" in [metric_path] {\n\t\tmutate { add_field => { \"time_in_seconds\" => \"%{metric_value_string}\"} }\n\t\tmutate { gsub => [\"time_in_seconds\", \".0 secs\", \"\"] }\n\t\tmutate { convert => { \"time_in_seconds\""}

Here's the config file contents from that error above:

input { 
    jmx {
      path => "plugins/jmx"
      polling_frequency => 60
      type => "jmx"
      nb_thread => 4
    }   
}

filter {
    if "TimeSince" in [metric_path] or "Delay" in [metric_path] {
        mutate { add_field => { "time_in_seconds" => "%{metric_value_string}"} }
        mutate { gsub => ["time_in_seconds", ".0 secs", ""] }
        mutate { convert => { "time_in_seconds", "integer" } }
    }
}

output {
   stdout { codec => rubydebug }
}

No issues if I comment out the convert line..

Am I going about this the wrong way? I am newer to this stack. If this was C# / SQL I'd simply do a replace + cast / convert. Am I doing this in the wrong place?

Here's the STDOUT when I comment out the convert line so you can see actual data:

C:\Elastic\logstash-5.2.2\bin>cls
    C:\Elastic\logstash-5.2.2\bin>logstash -f config/logstash.conf --config.reload.automatic 
    JAVA_OPTS was set to [ -Dlog4j.configurationFile=C:\Elastic\logstash-5.2.2\config\log4j2.properties     -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="C:\Elastic\logstash-5.2.2/heapdump.hprof"]. Logstash will trust these options, and not set any defaults that it might usually set
    [2017-09-14T14:23:13,456][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
    [2017-09-14T14:23:13,568][INFO ][logstash.inputs.jmx      ] Create queue dispatching JMX requests to threads
    [2017-09-14T14:23:13,573][INFO ][logstash.inputs.jmx      ] Compile regexp for group alias object replacement
    [2017-09-14T14:23:13,574][INFO ][logstash.pipeline        ] Pipeline main started
    [2017-09-14T14:23:13,576][INFO ][logstash.inputs.jmx      ] Initialize 4 threads for JMX metrics collection
    [2017-09-14T14:23:13,648][INFO ][logstash.inputs.jmx      ] Loading configuration files in path {:path=>"plugins/jmx"}
    [2017-09-14T14:23:13,743][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
    {
                       "path" => "plugins/jmx",
                "environment" => "TEST",
                 "@timestamp" => 2017-09-14T18:23:14.154Z,
                   "@version" => "1",
                       "host" => "MY HOST",
                "metric_path" => "xxStatus",
                       "type" => "jmx",
        "metric_value_string" => "idle"
    }
    {
                       "path" => "plugins/jmx",
            "time_in_seconds" => "191",
                "environment" => "TEST",
                 "@timestamp" => 2017-09-14T18:23:14.200Z,
                   "@version" => "1",
                       "host" => "MY HOST",
                "metric_path" => "xxTimeSincexx",
                       "type" => "jmx",
        "metric_value_string" => "191.0 secs"
    }

(Chris Caputo) #2

It was bad syntax on mutate.

According to [SOLVED] The filter plugin mutate-convert doesn't work in 5.0, the syntax should be:

filter {
	if "TimeSince" in [metric_path] or "Delay" in [metric_path] {
		mutate { add_field => { "time_in_seconds" => "%{metric_value_string}"} }
		mutate { gsub => ["time_in_seconds", ".0 secs", ""] }
		mutate { convert => ["time_in_seconds", "integer"] }
	}
}

instead of

filter {
    if "TimeSince" in [metric_path] or "Delay" in [metric_path] {
        mutate { add_field => { "time_in_seconds" => "%{metric_value_string}"} }
        mutate { gsub => ["time_in_seconds", ".0 secs", ""] }
        mutate { convert => { "time_in_seconds", "integer" } }
    }
}

The incorrect syntax is listed here @ https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert


(Eleanor Robbins) #3

As a newbie, I have to face different types of errors, in fact, I am in a learning process and already joined some famous and helpful forum like elastic.com so when I get some error or need any assignment help then I must visit here to get some useful answers.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.