@timestamp to unix epoch conversion in Logstash 7.0.1

I am attempting to convert @timestamp to unix epoch.
When running in a online parser:
https://repl.it/repls/FlatPristineEmulation it does what it supposed to do.


My test configuration:
input {stdin { }}

filter {
 grok { match => ["message", "%{GREEDYDATA:test_data}$" ]}
 mutate{ add_field => ["timestamp_unix","%{test_data}"] }
 ruby { code => "require 'date';event['epoch'] = DateTime.parse(event['timestamp_unix']).strftime('%s')" }
 mutate{ remove_field => ["timestamp_unix"] }

output {stdout { codec => rubydebug }}

Test stdin:

Debug output:

[DEBUG] 2019-05-15 12:01:11.341 [[main]>worker2] grok - Running grok filter {:event=>#<LogStash::Event:0x3483697>}
[DEBUG] 2019-05-15 12:01:11.350 [[main]>worker2] grok - Event now:  {:event=>#<LogStash::Event:0x3483697>}
[DEBUG] 2019-05-15 12:01:11.359 [[main]>worker2] decorators - filters/LogStash::Filters::Mutate: adding value to field {"field"=>"timestamp_unix", "value"=>["%{test_data}"]}
[ERROR] 2019-05-15 12:01:11.362 [[main]>worker2] ruby - Ruby exception occurred: undefined method `[]' for #<LogStash::Event:0x3483697>
[DEBUG] 2019-05-15 12:01:11.365 [[main]>worker2] mutate - filters/LogStash::Filters::Mutate: removing field {:field=>"timestamp_unix"}
       "message" => "2019-05-15T09:56:11.791Z",
     "test_data" => "2019-05-15T09:56:11.791Z",
          "tags" => [
        [0] "_rubyexception"
    "@timestamp" => 2019-05-15T10:01:11.236Z,
          "host" => "localhost",
      "@version" => "1"

Looks like I cannot access to the field somehow.
How it should be done in newest Logstash 7.0.1 release?

The way you access event fields inside Ruby blocks has changed in the recent Logstash versions.

Try this

filter {
 grok { match => ["message", "%{GREEDYDATA:test_data}$" ]}
 mutate{ add_field => ["timestamp_unix","%{test_data}"] }
 ruby { 
     init => "require 'date'"
     code => "event.set('epoch', DateTime.parse(event.get('timestamp_unix')).strftime('%s'))" 
 mutate{ remove_field => ["timestamp_unix"] }

output {stdout { codec => rubydebug }}

Seems to throw error:

[DEBUG] 2019-05-15 14:33:24.669 [Converge PipelineAction::Create<main>] javapipeline - Starting pipeline {:pipeline_id=>"main"}
warning: thread "[main]-pipeline-manager" terminated with exception (report_on_exception is true):
SyntaxError: (ruby filter code):3: syntax error, unexpected keyword_end

                     eval at org/jruby/RubyKernel.java:1061
                 register at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:59
                 register at org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56
         register_plugins at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191
                     each at org/jruby/RubyArray.java:1792
         register_plugins at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190
  maybe_setup_out_plugins at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:446
            start_workers at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:203
                      run at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:145
                    start at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:104
[ERROR] 2019-05-15 14:33:24.774 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[TRACE] 2019-05-15 14:33:24.786 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Converge results {:success=>false, :failed_actions=>["id: main, action_type: LogStash::PipelineAction::Create, message: Could not execute action: PipelineAction::Create<main>, action_result: false"], :successful_actions=>[]}

Oh right, I missed a parenthesis on the above code. Fixed it.

Wouldn't it be simpler to use to_f or to_i?

ruby { code => 'event.set( "epoch", event.get("@timestamp").to_f)' }

Thanks Badger.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.