Ruby error in adding millisecond to timestamp copy

Elastic Stack Logstash
1

Hi,

There seems to be an error in my ruby script, but I have no idea what it is, as the log is showing nothing... Please help!

part of config file:

date {
  match => ["date", "yyyy-MM-dd HH:mm:ss.SSS"]
}
grok {
  match => ["message", "checkName: %{WORD:checkName}, execution time: %{NUMBER:execution_duration}, path: %{NOTSPACE:step_path}"]
}
if [execution_duration] {
  mutate {
    convert => {"execution_duration" => "integer"}
    copy => {"@timestamp" => "end_time"}
  }
      ruby {
        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
      }
}

Error log:

[2018-08-20T10:16:46,779][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-08-20T10:17:02,759][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,770][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,781][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,783][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,845][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,851][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,863][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:02,965][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,116][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,127][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,143][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,145][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,192][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,198][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,215][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,223][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,245][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,249][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,273][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,277][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,283][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,288][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,292][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,297][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,297][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,304][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,324][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,329][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,341][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,351][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,396][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,441][ERROR][logstash.filters.ruby    ] Ruby exception occurred: 
[2018-08-20T10:17:03,456][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"", :error_class=>"LogStash::Json::GeneratorError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:28:in `jruby_dump'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:115:in `block in bulk'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:115:in `block in bulk'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:113:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:232:in `safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:118:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:479:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:478:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:430:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:in `block in start_workers'"]}
2

Use the math filter to do basic arithmetic on numeric fields in Logstash (in works on timestamps as well).

bin/logstash-plugin install logstash-filter-math

The docs have not made it to the main elastic docs yet. You can read the docs here

3

I added the following and still have error... Do I have to convert back the float to timestamp?

config file:

date {
  match => ["date", "yyyy-MM-dd HH:mm:ss.SSS"]
}
grok {
  match => ["message", "checkName: %{WORD:checkName}, execution time: %{NUMBER:execution_duration}, path: %{NOTSPACE:step_path}"]
}
if [execution_duration] {
  mutate {
    convert => {"execution_duration" => "integer"}
    copy => {"@timestamp" => "end_time"}
  }
#      ruby {
#        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
#      }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "MEM[0]"],
      ["add", "MEM[0]", "[end_time]", "[end_time]"]
    ]
  }
}
  }
4

Error:

[2018-08-20T11:22:29,550][ERROR][logstash.pipeline ] ne.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:332)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:332)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)",

5

"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x7a280dbd@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}

6

You don't have to convert the float to a timestamp because the Logstash Timestamp class has a + method that converts the Timestamp value to a Ruby Time instance then adds the float value to the Time instance and returns it. But then when a Ruby Time value is saved into an event it is converted into a LS Timestamp.

EDIT

The above is wrong, the Timestamp is converted to a Float via Ruby's Time class before the math filter adds the other Float to it - the result is stored in the event as a Float.

7

Damn and blast - I screwed up, the math filter is not thread safe for storing intermediate results in memory registers. I get wrong results when multiple events are processed on different worker threads.

You can use the event field to store intermediate results though with worker thread safety.

For reference, on LS 6.3.2 this config does not throw the errors you are seeing but it gives the wrong results when using .

input {
  generator {
    lines => ['2018-08-21 10:11:12|101', '2018-08-21 11:12:13|111', '2018-08-21 12:13:14|121']
    count => 1
  }
}

filter {
  dissect {
    mapping => { "[message]" => "%{adate}|%{millis}" }
    convert_datatype => {"[millis]" => "int"}
  }
  date {
    match => ["[adate]", "yyyy-MM-dd HH:mm:ss"]
    target => "[ats]"
  }
  math {
    calculate => [
      ["fdiv", "[millis]", 1000, "[fmillis]"],
      ["add", "[ats]", "[fmillis]", "[end_time]"]
    ]
  }
}

Gives:

{
      "sequence" => 0,
           "ats" => 2018-08-21T11:13:14.000Z,
    "@timestamp" => 2018-08-21T08:08:05.244Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "end_time" => 1534849994.121,
       "fmillis" => 0.121,
         "adate" => "2018-08-21 12:13:14",
       "message" => "2018-08-21 12:13:14|121",
        "millis" => 121
}
{
      "sequence" => 0,
           "ats" => 2018-08-21T10:12:13.000Z,
    "@timestamp" => 2018-08-21T08:08:05.243Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "end_time" => 1534846333.111,
       "fmillis" => 0.111,
         "adate" => "2018-08-21 11:12:13",
       "message" => "2018-08-21 11:12:13|111",
        "millis" => 111
}
{
      "sequence" => 0,
           "ats" => 2018-08-21T09:11:12.000Z,
    "@timestamp" => 2018-08-21T08:08:05.219Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "end_time" => 1534842672.101,
       "fmillis" => 0.101,
         "adate" => "2018-08-21 10:11:12",
       "message" => "2018-08-21 10:11:12|101",
        "millis" => 101
}
8

I used add_field instead of copy in replicating the timestamp. Now the error said the new value is not numeric or time... Is there anyway to switch that back and forth?

config file:

  mutate {
    convert => {"execution_duration" => "integer"}
#        copy => {"@timestamp" => "end_time"}
    add_field => { "end_time" => "%{@timestamp}" }
  }
#      ruby {
#        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
#      }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "[timediff]"],
      ["add", "[timediff]", "[end_time]", "[end_time]"]
    ]
  }
}

error:

[2018-08-21T13:46:26,819][WARN ][logstash.filters.mathcalculationelements.fieldelement] field value is not numeric or time {"field"=>"[end_time]", "value"=>"2018-08-21T17:46:02.766Z", "event"=>{"execution_duration"=>0, "message"=>"checkName: SafeBankingUI, execution time: 0, path: /banking/directDemoWelcome.do", "runningTime"=>"331719142", "hostname"=>"ws-server0", "host"=>{"name"=>"ws-server0"}, "UUID"=>"7bdfb01e-fa28-467c-a05c-faadf29888ad", "applicationName"=>"SafeBanking", "AQA-STM-ID"=>"05290684", "checkName"=>"SafeBankingUI", "step_path"=>"/banking/directDemoWelcome.do", "input"=>{"type"=>"log"}, "@version"=>"1", "fileLine"=>"LogUtils.java:139", "timediff"=>0.0, "@timestamp"=>2018-08-21T17:46:02.766Z, "date"=>"2018-08-21 13:46:02.766", "processId"=>"4964@ws-server0", "prospector"=>{"type"=>"log"}, "stmStepName"=>"SafeBanking_SafeBanking Login", "end_time"=>"2018-08-21T17:46:02.766Z", "source"=>"c:\\var\\logs\\oob-demo\\safebanking-sb201.log", "method"=>"endCheckPoint", "exception"=>"-", "log"=>"c.d.f.logging.filter.LogFilter:139", "serviceName"=>"SafeBankingUI", "offset"=>114312936, "thread"=>"WebContainer : 0", "fields"=>{"host_ip"=>"192.168.1.50"}, "beat"=>{"version"=>"6.3.2", "name"=>"ws-server0", "hostname"=>"ws-server0"}, "userId"=>"-", "tags"=>["safebanking", "beats_input_codec_plain_applied"], "level"=>"INFO ", "sessionId"=>"-"}}

9

Solved. Apparently I need to switch it back and forth...

  mutate {
    convert => {"execution_duration" => "integer"}
#        copy => {"@timestamp" => "end_time"}
    add_field => { "end_time" => "%{@timestamp}" }
  }
  date {
    match => ["end_time", "ISO8601"]
    target => ["end_time"]
  }
#      ruby {
#        code => 'event.set("end_time", LogStash::Timestamp.new(Time.at(event.get("end_time").to_f+(event.get("execution_duration").to_f/1000.00))))'
#      }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "[timediff]"],
      ["add", "[timediff]", "[end_time]", "[end_time]"]
    ]
  }
  date {
    match => ["end_time", "UNIX"]
    target => ["end_time"]
  }
10

Why can't you work directly with the @timestamp in the math filter, as you are making a copy anyway?


  mutate {
    convert => {"execution_duration" => "integer"}
  }
  math {
    calculate => [
      ["fdiv", "[execution_duration]", 1000, "[timediff]"],
      ["add", "[timediff]", "[@timestamp]", "[end_time]"]
    ]
  }
  date {
    match => ["end_time", "UNIX"]
    target => ["end_time"]
  }
1 Like
11

Ya that simplifies things a lot lol. Thanks!

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.