Ruby error in adding millisecond to timestamp copy

perryparktung · August 20, 2018, 2:30pm

Hi,

There seems to be an error in my ruby script, but I have no idea what it is, as the log is showing nothing... Please help!

part of config file:

date {
  match => ["date", "yyyy-MM-dd HH:mm:ss.SSS"]
}
grok {
  match => ["message", "checkName: %{WORD:checkName}, execution time: %{NUMBER:execution_duration}, path: %{NOTSPACE:step_path}"]
}
if [execution_duration] {
  mutate {
    convert => {"execution_duration" => "integer"}
    copy => {"@timestamp" => "end_time"}
  }
      ruby {
        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
      }
}

Error log:

[2018-08-20T10:16:46,779][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-08-20T10:17:02,759][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,770][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,781][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,783][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,845][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,851][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,863][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,965][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,116][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,127][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,143][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,145][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,192][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,198][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,215][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,223][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,245][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,249][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,273][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,277][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,283][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,288][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,292][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,297][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,297][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,304][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,324][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,329][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,341][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,351][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,396][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,441][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,456][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"", :error_class=>"LogStash::Json::GeneratorError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:28:in `jruby_dump'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:115:in `block in bulk'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:115:in `block in bulk'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:113:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:232:in `safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:118:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:479:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:478:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:430:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:in `block in start_workers'"]}

guyboertje · August 20, 2018, 2:54pm

Use the math filter to do basic arithmetic on numeric fields in Logstash (in works on timestamps as well).

bin/logstash-plugin install logstash-filter-math

The docs have not made it to the main elastic docs yet. You can read the docs here

perryparktung · August 20, 2018, 3:30pm

I added the following and still have error... Do I have to convert back the float to timestamp?

config file:

date {
  match => ["date", "yyyy-MM-dd HH:mm:ss.SSS"]
}
grok {
  match => ["message", "checkName: %{WORD:checkName}, execution time: %{NUMBER:execution_duration}, path: %{NOTSPACE:step_path}"]
}
if [execution_duration] {
  mutate {
    convert => {"execution_duration" => "integer"}
    copy => {"@timestamp" => "end_time"}
  }
#      ruby {
#        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
#      }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "MEM[0]"],
      ["add", "MEM[0]", "[end_time]", "[end_time]"]
    ]
  }
}
  }

perryparktung · August 20, 2018, 3:32pm

Error:

[2018-08-20T11:22:29,550][ERROR][logstash.pipeline ] ne.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:332)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:332)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)",

perryparktung · August 20, 2018, 3:32pm

"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x7a280dbd@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}

guyboertje · August 20, 2018, 9:14pm

You don't have to convert the float to a timestamp because the Logstash Timestamp class has a + method that converts the Timestamp value to a Ruby Time instance then adds the float value to the Time instance and returns it. But then when a Ruby Time value is saved into an event it is converted into a LS Timestamp.

EDIT

The above is wrong, the Timestamp is converted to a Float via Ruby's Time class before the math filter adds the other Float to it - the result is stored in the event as a Float.

guyboertje · August 21, 2018, 8:10am

Damn and blast - I screwed up, the math filter is not thread safe for storing intermediate results in memory registers. I get wrong results when multiple events are processed on different worker threads.

You can use the event field to store intermediate results though with worker thread safety.

For reference, on LS 6.3.2 this config does not throw the errors you are seeing but it gives the wrong results when using .

input {
  generator {
    lines => ['2018-08-21 10:11:12|101', '2018-08-21 11:12:13|111', '2018-08-21 12:13:14|121']
    count => 1
  }
}

filter {
  dissect {
    mapping => { "[message]" => "%{adate}|%{millis}" }
    convert_datatype => {"[millis]" => "int"}
  }
  date {
    match => ["[adate]", "yyyy-MM-dd HH:mm:ss"]
    target => "[ats]"
  }
  math {
    calculate => [
      ["fdiv", "[millis]", 1000, "[fmillis]"],
      ["add", "[ats]", "[fmillis]", "[end_time]"]
    ]
  }
}

Gives:

{
      "sequence" => 0,
           "ats" => 2018-08-21T11:13:14.000Z,
    "@timestamp" => 2018-08-21T08:08:05.244Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "end_time" => 1534849994.121,
       "fmillis" => 0.121,
         "adate" => "2018-08-21 12:13:14",
       "message" => "2018-08-21 12:13:14|121",
        "millis" => 121
}
{
      "sequence" => 0,
           "ats" => 2018-08-21T10:12:13.000Z,
    "@timestamp" => 2018-08-21T08:08:05.243Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "end_time" => 1534846333.111,
       "fmillis" => 0.111,
         "adate" => "2018-08-21 11:12:13",
       "message" => "2018-08-21 11:12:13|111",
        "millis" => 111
}
{
      "sequence" => 0,
           "ats" => 2018-08-21T09:11:12.000Z,
    "@timestamp" => 2018-08-21T08:08:05.219Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "end_time" => 1534842672.101,
       "fmillis" => 0.101,
         "adate" => "2018-08-21 10:11:12",
       "message" => "2018-08-21 10:11:12|101",
        "millis" => 101
}

perryparktung · August 21, 2018, 5:53pm

I used add_field instead of copy in replicating the timestamp. Now the error said the new value is not numeric or time... Is there anyway to switch that back and forth?

config file:

  mutate {
    convert => {"execution_duration" => "integer"}
#        copy => {"@timestamp" => "end_time"}
    add_field => { "end_time" => "%{@timestamp}" }
  }
#      ruby {
#        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
#      }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "[timediff]"],
      ["add", "[timediff]", "[end_time]", "[end_time]"]
    ]
  }
}

error:

[2018-08-21T13:46:26,819][WARN ][logstash.filters.mathcalculationelements.fieldelement] field value is not numeric or time {"field"=>"[end_time]", "value"=>"2018-08-21T17:46:02.766Z", "event"=>{"execution_duration"=>0, "message"=>"checkName: SafeBankingUI, execution time: 0, path: /banking/directDemoWelcome.do", "runningTime"=>"331719142", "hostname"=>"ws-server0", "host"=>{"name"=>"ws-server0"}, "UUID"=>"7bdfb01e-fa28-467c-a05c-faadf29888ad", "applicationName"=>"SafeBanking", "AQA-STM-ID"=>"05290684", "checkName"=>"SafeBankingUI", "step_path"=>"/banking/directDemoWelcome.do", "input"=>{"type"=>"log"}, "@version"=>"1", "fileLine"=>"LogUtils.java:139", "timediff"=>0.0, "@timestamp"=>2018-08-21T17:46:02.766Z, "date"=>"2018-08-21 13:46:02.766", "processId"=>"4964@ws-server0", "prospector"=>{"type"=>"log"}, "stmStepName"=>"SafeBanking_SafeBanking Login", "end_time"=>"2018-08-21T17:46:02.766Z", "source"=>"c:\\var\\logs\\oob-demo\\safebanking-sb201.log", "method"=>"endCheckPoint", "exception"=>"-", "log"=>"c.d.f.logging.filter.LogFilter:139", "serviceName"=>"SafeBankingUI", "offset"=>114312936, "thread"=>"WebContainer : 0", "fields"=>{"host_ip"=>"192.168.1.50"}, "beat"=>{"version"=>"6.3.2", "name"=>"ws-server0", "hostname"=>"ws-server0"}, "userId"=>"-", "tags"=>["safebanking", "beats_input_codec_plain_applied"], "level"=>"INFO ", "sessionId"=>"-"}}

perryparktung · August 21, 2018, 9:27pm

Solved. Apparently I need to switch it back and forth...

  mutate {
    convert => {"execution_duration" => "integer"}
#        copy => {"@timestamp" => "end_time"}
    add_field => { "end_time" => "%{@timestamp}" }
  }
  date {
    match => ["end_time", "ISO8601"]
    target => ["end_time"]
  }
#      ruby {
#        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
#      }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "[timediff]"],
      ["add", "[timediff]", "[end_time]", "[end_time]"]
    ]
  }
  date {
    match => ["end_time", "UNIX"]
    target => ["end_time"]
  }

guyboertje · August 22, 2018, 9:38am

Why can't you work directly with the @timestamp in the math filter, as you are making a copy anyway?


  mutate {
    convert => {"execution_duration" => "integer"}
  }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "[timediff]"],
      ["add", "[timediff]", "[@timestamp]", "[end_time]"]
    ]
  }
  date {
    match => ["end_time", "UNIX"]
    target => ["end_time"]
  }

perryparktung · August 22, 2018, 5:36pm

Ya that simplifies things a lot lol. Thanks!

system · September 19, 2018, 5:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.