Mutate filter throws NumberFormat Exception for no reason

Elastic Stack Logstash
1

I recently migrated from 3.1 to 5.1 and wanted to continue to use my logstash filter/output configuration. Since the event API changed I adopted those but soon realised that the mutate filter is suddenly throwing excpetions that haven't been there before!

In 3.1 I had a filter that looked like this:

if ( [businessTransactions][type] == 1 ) {
  mutate {
    add_field => {
      "clientTime" => '%{[businessTransactions][occurrences][clientTime]}'
      "serverTime" => '%{[businessTransactions][occurrences][serverTime]}'
    }
  }
}

This gave me float values in the fields that were nicely written to influxDB afterwards.
First I realised that before there was nice automatic conversion to float and with 5.1 these float values were written as string now. So i added a mutate convert filter

if ( [businessTransactions][type] == 1 ) {
  mutate {
    add_field => {
      "clientTime" => '%{[businessTransactions][occurrences][clientTime]}'
      "serverTime" => '%{[businessTransactions][occurrences][serverTime]}'
    }
  }
  mutate {
    convert => {
      "serverTime" => "float"
      "clientTime" => "float"
    }
  }
}

Doing so 'sometimes' gives me a Exception in syslog. I can guarantee that the values in clientTime are always numbers and no strings. This is the top stacktrace:

Exception in thread "[main]>worker3" java.lang.NumberFormatException: For input string: "clientTime"
#011at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
#011at java.lang.Integer.parseInt(Integer.java:580)
#011at java.lang.Integer.parseInt(Integer.java:615)
#011at org.logstash.Accessors.fetch(Accessors.java:130)
#011at org.logstash.Accessors.get(Accessors.java:20)
#011at org.logstash.Event.getUnconvertedField(Event.java:160)
#011at org.logstash.Event.getField(Event.java:150)
#011at org.logstash.KeyNode.evaluate(KeyNode.java:26)
#011at org.logstash.StringInterpolation.evaluate(StringInterpolation.java:47)
#011at org.logstash.Event.sprintf(Event.java:294)
#011at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_sprintf(JrubyEventExtLibrary.java:200)
...
#011at rubyjit.LogStash::Filters::Mutate$$filter_7ece4ecf2c9d8324f6c4933fda087138f0104c701442407170.__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.1.3/lib/logstash/filters/mutate.rb:216)
#011at rubyjit.LogStash::Filters::Mutate$$filter_7ece4ecf2c9d8324f6c4933fda087138f0104c701442407170.__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.1.3/lib/logstash/filters/mutate.rb)
#011at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:201)
#011at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
#011at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
#011at rubyjit.LogStash::Filters::Base$$do_filter_8e8403dcfdf01a35ffca12ed35ec4e79455489071442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
#011at rubyjit.LogStash::Filters::Base$$do_filter_8e8403dcfdf01a35ffca12ed35ec4e79455489071442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb)
#011at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:201)

So I added the conversion and field creation directly in a ruby filter:

if ( [businessTransactions][type] == 1 ) {
  ruby {
    code => "
      unless event.get('[businessTransactions][occurrences][clientTime]').nil?
      event.set('clientTime',sprintf('%.4f',event.get('[businessTransactions][occurrences][clientTime]')).to_f)
    end
      unless event.get('[businessTransactions][occurrences][serverTime]').nil?
      event.set('serverTime',sprintf('%.4f',event.get('[businessTransactions][occurrences][serverTime]')).to_f)
    end
    "
  }
}

This seems to work but I have other filters where I now get NumberFormat Exceptions that do not make sense at all. e.g. this mutate:

mutate {
  add_field => {
    "tagging_values" => "%{[businessTransactions][occurrences][dimensions]}"
    "application" => "%{[businessTransactions][application]}"
    "businesstransaction" => "%{[businessTransactions][name]}"
    "systemprofile" => "%{[businessTransactions][systemProfile]}"
    "project" => "%{[businessTransactions][systemProfile]}"
    "responsetime" => '%{[businessTransactions][occurrences][responseTime]}'
    "time" => "%{[businessTransactions][occurrences][startTime]}"
    "resultmeasure_names" => "%{[businessTransactions][measureNames]}"
    "resultmeasure_values" => "%{[businessTransactions][occurrences][values]}"
  }
}

It now also throws an exception after I've fixed the above.:

Exception in thread "[main]>worker1" java.lang.NumberFormatException: For input string: "dimensions"
#011at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
#011at java.lang.Integer.parseInt(Integer.java:580)
#011at java.lang.Integer.parseInt(Integer.java:615)
#011at org.logstash.Accessors.fetch(Accessors.java:130)
#011at org.logstash.Accessors.get(Accessors.java:20)
#011at org.logstash.Event.getUnconvertedField(Event.java:160)
#011at org.logstash.Event.getField(Event.java:150)
#011at org.logstash.KeyNode.evaluate(KeyNode.java:26)

The data I'm feeding into logstash is sane and correct, as soon as I hook up my event source to logstash 3.1 everything works fine.

Any ideas? I've seen that there are other posts about mutate filter problems...

Thanks,
Reinhard

2

Please show an example event, preferably using a stdout { codec => rubydebug } output.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.