@timestamp to unix epoch conversion in Logstash 7.0.1

Elastic Stack Logstash
1

Hello!
I am attempting to convert @timestamp to unix epoch.
When running in a online parser:
https://repl.it/repls/FlatPristineEmulation it does what it supposed to do.

ruby_compile

My test configuration:
input {stdin { }}

filter {
 grok { match => ["message", "%{GREEDYDATA:test_data}$" ]}
 mutate{ add_field => ["timestamp_unix","%{test_data}"] }
 ruby { code => "require 'date';event['epoch'] = DateTime.parse(event['timestamp_unix']).strftime('%s')" }
 mutate{ remove_field => ["timestamp_unix"] }
}

output {stdout { codec => rubydebug }}

Test stdin:
2019-05-15T09:56:11.791Z

Debug output:

[DEBUG] 2019-05-15 12:01:11.341 [[main]>worker2] grok - Running grok filter {:event=>#<LogStash::Event:0x3483697>}
[DEBUG] 2019-05-15 12:01:11.350 [[main]>worker2] grok - Event now:  {:event=>#<LogStash::Event:0x3483697>}
[DEBUG] 2019-05-15 12:01:11.359 [[main]>worker2] decorators - filters/LogStash::Filters::Mutate: adding value to field {"field"=>"timestamp_unix", "value"=>["%{test_data}"]}
[ERROR] 2019-05-15 12:01:11.362 [[main]>worker2] ruby - Ruby exception occurred: undefined method `[]' for #<LogStash::Event:0x3483697>
[DEBUG] 2019-05-15 12:01:11.365 [[main]>worker2] mutate - filters/LogStash::Filters::Mutate: removing field {:field=>"timestamp_unix"}
{
       "message" => "2019-05-15T09:56:11.791Z",
     "test_data" => "2019-05-15T09:56:11.791Z",
          "tags" => [
        [0] "_rubyexception"
    ],
    "@timestamp" => 2019-05-15T10:01:11.236Z,
          "host" => "localhost",
      "@version" => "1"
}

Looks like I cannot access to the field somehow.
How it should be done in newest Logstash 7.0.1 release?

2

The way you access event fields inside Ruby blocks has changed in the recent Logstash versions.

Try this

filter {
 grok { match => ["message", "%{GREEDYDATA:test_data}$" ]}
 mutate{ add_field => ["timestamp_unix","%{test_data}"] }
 ruby { 
     init => "require 'date'"
     code => "event.set('epoch', DateTime.parse(event.get('timestamp_unix')).strftime('%s'))" 
 }
 mutate{ remove_field => ["timestamp_unix"] }
}

output {stdout { codec => rubydebug }}
3

Seems to throw error:

[DEBUG] 2019-05-15 14:33:24.669 [Converge PipelineAction::Create<main>] javapipeline - Starting pipeline {:pipeline_id=>"main"}
warning: thread "[main]-pipeline-manager" terminated with exception (report_on_exception is true):
SyntaxError: (ruby filter code):3: syntax error, unexpected keyword_end

                     eval at org/jruby/RubyKernel.java:1061
                 register at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:59
                 register at org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56
         register_plugins at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191
                     each at org/jruby/RubyArray.java:1792
         register_plugins at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190
  maybe_setup_out_plugins at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:446
            start_workers at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:203
                      run at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:145
                    start at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:104
[ERROR] 2019-05-15 14:33:24.774 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[TRACE] 2019-05-15 14:33:24.786 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Converge results {:success=>false, :failed_actions=>["id: main, action_type: LogStash::PipelineAction::Create, message: Could not execute action: PipelineAction::Create<main>, action_result: false"], :successful_actions=>[]}
4

Oh right, I missed a parenthesis on the above code. Fixed it.

5

Wouldn't it be simpler to use to_f or to_i?

ruby { code => 'event.set( "epoch", event.get("@timestamp").to_f)' }
6

Thanks Badger.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.