Convert the @timestamp field

Saravana37 · May 25, 2021, 7:56am

Hello All ,

I am trying to convert the @timestamp field to the required format.
@timestamp shows the field as : 2021-05-25T07:34:08.137Z and I want the value to be converted in to this format : 05/25/2021 07:34:08 . I have tried the below date filter format.
date { match => [ "@timestamp", "MM/dd/YYYY HH:mm:ss", "YYYY-MM-ddTHH:mm:ss.sssZ", "ISO8601" ] }

And the error shows as below. Please help.

E:\ELK\logstash\bin>logstash.bat -f E:\ELK\logstash\config\conf\servermgr.conf
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/E:/ELK/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.Console.cs
WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to E:/ELK/logstash/logs which is now configured via log4j2.properties
[2021-05-25T09:52:11,204][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-05-25T09:52:11,219][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.4"}
[2021-05-25T09:52:16,575][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalArgumentException", :message=>"Illegal pattern component: T", :backtrace=>["org.joda.time.format.DateTimeFormat.parsePatternTo(org/joda/time/format/DateTimeFormat.java:566)", "org.joda.time.format.DateTimeFormat.createFormatterForPattern(org/joda/time/format/DateTimeFormat.java:687)", "org.joda.time.format.DateTimeFormat.forPattern(org/joda/time/format/DateTimeFormat.java:177)", "org.logstash.filters.parser.JodaParser.<init>(org/logstash/filters/parser/JodaParser.java:58)", "org.logstash.filters.parser.TimestampParserFactory.makeParser(org/logstash/filters/parser/TimestampParserFactory.java:60)", "org.logstash.filters.parser.TimestampParserFactory.makeParser(org/logstash/filters/parser/TimestampParserFactory.java:69)", "org.logstash.filters.DateFilter.acceptFilterConfig(org/logstash/filters/DateFilter.java:66)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:485)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:340)", "E_3a_.ELK.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_date_minus_3_dot_1_dot_9.lib.logstash.filters.date.initialize(E:/ELK/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-date-3.1.9/lib/logstash/filters/date.rb:185)", "org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2563)", "org.jruby.RubyArray.map(org/jruby/RubyArray.java:2577)", "org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)", "E_3a_.ELK.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_date_minus_3_dot_1_dot_9.lib.logstash.filters.date.initialize(E:/ELK/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-date-3.1.9/lib/logstash/filters/date.rb:184)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:894)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:798)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.plugins.PluginFactoryExt$Plugins.filter_delegator(org/logstash/plugins/PluginFactoryExt.java:78)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:248)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:184)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.pipeline.plugin(E:/ELK/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "RUBY.initialize((eval):199)", "org.jruby.RubyKernel.evalCommon(org/jruby/RubyKernel.java:1099)", "org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:1061)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval.call(org/jruby/RubyKernel$INVOKER$s$0$3$eval.gen)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.pipeline.initialize(E:/ELK/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.pipeline.initialize(E:/ELK/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.execute(E:/ELK/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:266)", "org.jruby.RubyProc$INVOKER$i$0$0$call.call(org/jruby/RubyProc$INVOKER$i$0$0$call.gen)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.agent.exclusive(E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:96)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:165)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(org/jruby/ext/thread/Mutex$INVOKER$i$0$0$synchronize.gen)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.agent.exclusive(E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:96)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0$__VARARGS__(E_3a_/ELK/logstash/logstash_minus_core/lib/logstash/E:/ELK/logstash/logstash-core/lib/logstash/agent.rb)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.execute(E:/ELK/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(E_3a_/ELK/logstash/logstash_minus_core/lib/logstash/pipeline_action/E:/ELK/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "E_3a_.ELK.logstash.logstash_minus_core.lib.logstash.agent.converge_state(E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:334)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalArgumentException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:109
             add at org/logstash/execution/ConvergeResultExt.java:37
  converge_state at E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:347
[2021-05-25T09:52:16,653][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalArgumentException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:347:in `block in converge_state'"]}
[2021-05-25T09:52:16,855][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalArgumentException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:347:in `block in converge_state'"]}
[2021-05-25T09:52:16,871][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Badger · May 25, 2021, 2:44pm

You need to quote literal characters

"YYYY-MM-dd'T'HH:mm:ss.sss'Z'"

Saravana37 · May 26, 2021, 7:57am

Hello @Badger ,

Thank you so much for your response. I am trying to convert the @timestamp field and getting _dateparsefailure . I tried with both below code but no luck. Please help.

date { match => [ "@timestamp", "MM/dd/YYYY HH:mm:ss", "YYYY-MM-dd'T'HH:mm:ss.sss'Z'", "ISO8601" ] }

also tried ,

date { match => [ "@timestamp", "MM/dd/YYYY HH:mm:ss", "MMM dd, YYYY @ HH:mm:ss.sss", "ISO8601" ] }

Thanks Again

Badger · May 26, 2021, 2:24pm

Your [@timestamp] field is already a date. Why are you trying to parse it again?

Saravana37 · May 27, 2021, 12:41pm

I want to print it in the required format . How can i do that ?

Badger · May 27, 2021, 3:28pm

elasticsearch stores a date as a long (milliseconds since the epoch). kibana formats this. You can reconfigure kibana to format it differently.

If you want to store it in elasticsearch as a string with a different format then use ruby and strftime.

Saravana37 · June 1, 2021, 5:26am

Thank you @Badger . It worked.

system · June 29, 2021, 5:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.