Regarding Date filter format

Elastic Stack Logstash
1

Hi team,

I have a field having value as
"updatedDate" => 2019-12-31T00:27:17.356Z
I am trying to convert date format to
updatedDate" => 2019-12-31

I am using below filter.But getting datetime parse error:

date {
match => [ "updatedDate", "yyyy-MM-dd" ]
target => "updatedDate"
}

could you please help on this?

Thanks,
Suresh.

2

Hello @suresh_u

You can try like the below which would work

filter
{
ruby {
        code => '
            t = event.get("updatedDate")
            varDateStringToDate = DateTime.strptime(t,"%Y-%m-%dT%H:%M:%S")
            event.set("updatedDate", varDateStringToDate.strftime("%Y-%m-%d"))
        '
    }
}

Keep Posted!!! Thanks !!!

3

I am getting rubyexception @sudhagar_ramesh

"tags" => [
[0] "_rubyexception"
],

filter {
ruby {
code => '
t = event.get("updatedDate")
varDateStringToDate = DateTime.strptime(t,"%Y-%m-%dT%H:%M:%S%z")
event.set("updatedDate", varDateStringToDate.strftime("%Y-%m-%d"))
'
}
}

updatedDate is not converted to required format.

Thanks.

4

@suresh_u Reproducing the issue. Give me sometime

5

Hello @suresh_u

Updated the code. Please check and feedback

6

sure @sudhagar_ramesh

7

could you please share the updated code? @sudhagar_ramesh

8

Edited the code in my 1st reply , check this post 2 @suresh_u

9

Getting the same rubyException.

I am sharing the log:

[2022-07-01T12:26:13,807][ERROR][logstash.filters.ruby ][main][0793217f3704ffec438d3766e047cdb433cf2fd9ef5ebdd7a4afa73c63a6d898] Ruby exception occurred: no implicit conversion of LogStash::Timestamp into String {:class=>"TypeError", :backtrace=>["org/jruby/ext/date/RubyDate.java:1684:in _strptime'", "org/jruby/ext/date/RubyDateTime.java:551:in _strptime'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/date.rb:714:in strptime'", "(ruby filter code):4:in block in filter_method'", "C:/Users/Suresh_Ummiti/WorkingDirectory/Softwares/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in inline_script'", "C:/Users/Suresh_Ummiti/WorkingDirectory/Softwares/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in filter'", "C:/Users/Suresh_Ummiti/WorkingDirectory/Softwares/logstash-7.17.4/logstash-core/lib/logstash/filters/base.rb:159:in do_filter'", "C:/Users/Suresh_Ummiti/WorkingDirectory/Softwares/logstash-7.17.4/logstash-core/lib/logstash/filters/base.rb:178:in block in multi_filter'", "org/jruby/RubyArray.java:1821:in each'", "C:/Users/Suresh_Ummiti/WorkingDirectory/Softwares/logstash-7.17.4/logstash-core/lib/logstash/filters/base.rb:175:in multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in multi_filter'", "C:/Users/Suresh_Ummiti/WorkingDirectory/Softwares/logstash-7.17.4/logstash-core/lib/logstash/java_pipeline.rb:300:in block in start_workers'"]}

Could you please check @sudhagar_ramesh?

10

@suresh_u
As that field is timestamp , we can't change format it like string using strftime thats the reason for ruby exception.
Also date filter which you used is for converting strings to timestamps only.

Hence will use the grok pattern to get only the data from timestamp

filter
{
grok
{
match => {"message" => ["updatedDate" , "%{GREEDYDATA:updateDate}T%{GREEDYDATA:time}"]}
}
}

Keep Posted !!! Thanks!!!

1 Like
11

thanks for the update. Let me try

12

thanks a lot @sudhagar_ramesh , it is working.

1 Like
13

Hi @sudhagar_ramesh , one more POC.

In my oracle DB, I have with timestamp like this: "createdDate" => 2022-04-02T17:00:44.339Z
But in my previous elasticsearch index, having createdDate as "createdDate": "2022-04-02T17:00:44Z"
How can we change the format in Logstash configuration?

Thanks
Suresh.

14

In that case try the ruby code if still facing issue, please raise a new case for assistance.

15

sure @sudhagar_ramesh . Thanks a lot

16

There is an open issue for this. The standard fix is to mutate+convert the timestamp into a string.

17

thanks a lot @Badger. I will check this out.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.