Logstash to convert scientific notation to float

[the_elkguy (Rehan)]

Hello,

I'm using logstash ruby filter to convert scientific notation from string to float which seem to be not working. I have used GROK to extract the scientific notation (1.989e-04) from the message and now converting to real number (0.0001989).
The values i get is just 0

Here is my code.

        ruby {
            code => "
                require 'bigdecimal'
                event.set('sourceOffset', ('%.20f' % (BigDecimal.new event.get('sourceOffset'))).sub(/\.?0*$/, '').to_f)
            "

Also the logstash error:

[2022-10-10T09:09:08,123][ERROR][logstash.filters.ruby ] Ruby exception occurred: invalid value for BigDecimal(): "-" {:class=>"ArgumentError", :backtrace=>["org/jruby/ext/bigdecimal/RubyBigDecimal.java:716:in new'", "(ruby filter code):4:in block in filter_method'", "/apps/elk/logstash-7.17.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in inline_script'", "logstash-7.17.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in filter'", "logstash-7.17.1/logstash-core/lib/logstash/filters/base.rb:159:in do_filter'", "logstash-7.17.1/logstash-core/lib/logstash/filters/base.rb:178:in block in multi_filter'", "org/jruby/RubyArray.java:1821:in each'", "logstash-7.17.1/logstash-core/lib/logstash/filters/base.rb:175:in multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in multi_filter'", "/logstash-7.17.1/logstash-core/lib/logstash/java_pipeline.rb:299:in block in start_workers'"]}

Appreciate some help.
thanks,

[Badger]

If your [sourceOffset] field contains "1.989e-04" then your ruby filter will produce

"sourceOffset" => 0.0001989,

However, you have no exception handling. The Ruby exception occurred: invalid value for BigDecimal(): "-" occurs if the [sourceOffset] field contains -.

You could use try/catch or test whether [sourceOffset] is in scientific notation using a regexp.

[Rios]

Have you tried with mutate convert?

mutate {
        convert => {
          "sourceOffset" => "float"
        }
}

[the_elkguy (Rehan)]

@Badger

Yes that's working. I had to change to GROK patterns as %{USERNAME:sourceoffset}
then i ran the Ruby without the (.to_f) which worked.
the values are now coming as String.
Ruby failed to convert that to Float and so many error;s about the exceptions.

@Rios

I also treid to convert that to float, that actually shows back the scientific notation value again. so i removed that.

The only issue now is how do I convert it to Number type.

[Rios]

Number as integer or number as type?
If is as the type, it's already, and you have recreate mappings in Kibana.

[the_elkguy (Rehan)]

@Rios

Apparently its not
Ruby is converting as String.

[Rios]

When is converted to integer or float it's number. Use debug, will see values.

[the_elkguy (Rehan)]

My ruby is converting the scientific notation in string but i need in float
i need help here,
Mutate filter convert is showing back the scientific notation in number type # ( -1.038e-06 ) but not in decimal -0.000001038.

 ruby {
                        code => "
                        require 'bigdecimal'
                        event.set('chronyoffset', ('%.20f' % (BigDecimal.new event.get('chronyoffset'))).sub(/\.?0*$/, ''))
                        "
                        }

Please provide me correct ruby code to fix this.