Logstash to convert scientific notation to float


I'm using logstash ruby filter to convert scientific notation from string to float which seem to be not working. I have used GROK to extract the scientific notation (1.989e-04) from the message and now converting to real number (0.0001989).
The values i get is just 0

Here is my code.

        ruby {
            code => "
                require 'bigdecimal'
                event.set('sourceOffset', ('%.20f' % (BigDecimal.new event.get('sourceOffset'))).sub(/\.?0*$/, '').to_f)

Also the logstash error:

[2022-10-10T09:09:08,123][ERROR][logstash.filters.ruby ] Ruby exception occurred: invalid value for BigDecimal(): "-" {:class=>"ArgumentError", :backtrace=>["org/jruby/ext/bigdecimal/RubyBigDecimal.java:716:in new'", "(ruby filter code):4:in block in filter_method'", "/apps/elk/logstash-7.17.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in inline_script'", "logstash-7.17.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in filter'", "logstash-7.17.1/logstash-core/lib/logstash/filters/base.rb:159:in do_filter'", "logstash-7.17.1/logstash-core/lib/logstash/filters/base.rb:178:in block in multi_filter'", "org/jruby/RubyArray.java:1821:in each'", "logstash-7.17.1/logstash-core/lib/logstash/filters/base.rb:175:in multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in multi_filter'", "/logstash-7.17.1/logstash-core/lib/logstash/java_pipeline.rb:299:in block in start_workers'"]}

Appreciate some help.

If your [sourceOffset] field contains "1.989e-04" then your ruby filter will produce

"sourceOffset" => 0.0001989,

However, you have no exception handling. The Ruby exception occurred: invalid value for BigDecimal(): "-" occurs if the [sourceOffset] field contains -.

You could use try/catch or test whether [sourceOffset] is in scientific notation using a regexp.

Have you tried with mutate convert?

mutate {
        convert => {
          "sourceOffset" => "float"


Yes that's working. I had to change to GROK patterns as %{USERNAME:sourceoffset}
then i ran the Ruby without the (.to_f) which worked.
the values are now coming as String.
Ruby failed to convert that to Float and so many error;s about the exceptions.


I also treid to convert that to float, that actually shows back the scientific notation value again. so i removed that.

The only issue now is how do I convert it to Number type.

Number as integer or number as type?
If is as the type, it's already, and you have recreate mappings in Kibana.


Apparently its not
Ruby is converting as String.

When is converted to integer or float it's number. Use debug, will see values.

My ruby is converting the scientific notation in string but i need in float
i need help here,
Mutate filter convert is showing back the scientific notation in number type # ( -1.038e-06 ) but not in decimal -0.000001038.

 ruby {
                        code => "
                        require 'bigdecimal'
                        event.set('chronyoffset', ('%.20f' % (BigDecimal.new event.get('chronyoffset'))).sub(/\.?0*$/, ''))

Please provide me correct ruby code to fix this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.