# Datetime conversion error Logsatsh **URL:** \ **Category:** Logstash\ **Created:** [December 30, 2019, 8:09am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319 "2019-12-30T08:09:45Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 30, 2019, 8:09am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/1 "2019-12-30T08:09:46Z")
Hi Im having the below configuration in my logstash ```auto filter { xml { source => "result" target => "Incident" force_array => false } date{ match => ["sys_created_on","yyyy-MM-dd HH:mm:ss"] match => ["closed_at","yyyy-MM-dd HH:mm:ss"] match => ["opened_at","yyyy-MM-dd HH:mm:ss"] match => ["resolved_at","yyyy-MM-dd HH:mm:ss"] match => ["sys_updated_on","yyyy-MM-dd HH:mm:ss"] } } ``` Log Error: ```auto [2019-12-30T03:00:37,950][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2019-12-30T03:00:48,655][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.2.0"} [2019-12-30T03:00:52,724][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:snow, :exception=>"Java::JavaLang::IllegalArgumentException", :message=>"Illegal pattern component: c", :backtrace=>["org.joda.time.format.DateTimeFormat.parsePatternTo(org/joda/time/format/DateTimeFormat.java:566)", "org.joda.time.format.DateTimeFormat.createFormatterForPattern(org/joda/time/format/DateTimeFormat.java:687)", "org.joda.time.format.DateTimeFormat.forPattern(org/joda/time/format/DateTimeFormat.java:177)", "org.logstash.filters.parser.JodaParser.(org/logstash/filters/parser/JodaParser.java:58)", "org.logstash.filters.parser.TimestampParserFactory.makeParser(org/logstash/filters/parser/TimestampParserFactory.java:60)", "org.logstash.filters.parser.TimestampParserFactory.makeParser(org/logstash/filters/parser/TimestampParserFactory.java:69)", "org.logstash.filters.DateFilter.acceptFilterConfig(org/logstash/filters/DateFilter.java:66)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:485)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:340)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_date_minus_3_dot_1_dot_9.lib.logstash.filters.date.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-date-3.1.9/lib/logstash/filters/date.rb:185)", "org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2563)", "org.jruby.RubyArray.map(org/jruby/RubyArray.java:2577)", "org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_date_minus_3_dot_1_dot_9.lib.logstash.filters.date.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-date-3.1.9/lib/logstash/filters/date.rb:184)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:894)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:798)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.plugins.PluginFactoryExt$Plugins.filter_delegator(org/logstash/plugins/PluginFactoryExt.java:81)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:251)", "org.logstash.plugins.PluginFactoryExt$Plugins.buildFilter(org/logstash/plugins/PluginFactoryExt.java:160)", "org.logstash.config.ir.CompiledPipeline.setupFilters(org/logstash/config/ir/CompiledPipeline.java:133)", "org.logstash.config.ir.CompiledPipeline.(org/logstash/config/ir/CompiledPipeline.java:81)", "org.logstash.execution.JavaBasePipelineExt.initialize(org/logstash/execution/JavaBasePipelineExt.java:50)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(org/logstash/execution/JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.initialize(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]} [2019-12-30T03:00:53,295][FATAL][logstash.runner] An unexpected error occurred! {:error=>#`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:338:in `block in converge_state'"]} ``` Time field in my xml: ```auto 2019-12-18 03:25:37 ``` Kibana always recognizes my time fields as string if i do not apply any filters. Where am I going wrong? ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 30, 2019, 9:54am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/2 "2019-12-30T09:54:17Z")
Update: I had the error cleared with only having one column at a date filetr. > ``` > date{ > match => ["sys_created_on","yyyy-MM-dd HH:mm:ss"] > } > > ``` Yet my data is still recognized as a string in Kibana. Where am i wrong? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 30, 2019, 2:48pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/4 "2019-12-30T14:48:58Z")
When you have multiple instances of an option, the option is converted to an array. So ``` date{ match => ["sys_created_on","yyyy-MM-dd HH:mm:ss"] match => ["closed_at","yyyy-MM-dd HH:mm:ss"] match => ["opened_at","yyyy-MM-dd HH:mm:ss"] match => ["resolved_at","yyyy-MM-dd HH:mm:ss"] match => ["sys_updated_on","yyyy-MM-dd HH:mm:ss"] } ``` is equivalent to ``` date{ match => ["sys_created_on","yyyy-MM-dd HH:mm:ss", "closed_at","yyyy-MM-dd HH:mm:ss", "opened_at","yyyy-MM-dd HH:mm:ss", "resolved_at","yyyy-MM-dd HH:mm:ss", "sys_updated_on","yyyy-MM-dd HH:mm:ss"] } ``` A date filter takes the first entry in the array as a field name, and the rest of the entries as date patterns. But "closed\_at" is not a valid date pattern. In fact it starts with an "Illegal pattern component: c" You need to use five separate date filters. ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 30, 2019, 3:55pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/5 "2019-12-30T15:55:54Z")
@Badger, thank you for your response. I tried with one match only (sys\_created\_on), and it still isn't recognizing as a date. In kibana, when I'm importing the index freshly, it shows as string. Also, I'm not able to fully understand the issue with closed\_at. These are all columns in my incident table. Can you help me understand this better? Hope did I go about it? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 30, 2019, 4:43pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/6 "2019-12-30T16:43:12Z")
> [@katara](#): > > I tried with one match only (sys\_created\_on), and it still isn't recognizing as a date. By default, a date filter modifies the value of @timestamp. If you want it to convert sys\_created\_on to a timestamp then you must use the target option on the date filter. ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 30, 2019, 5:03pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/7 "2019-12-30T17:03:54Z")
Okay so of I understand this right, The value of sys\_created\_on will get saved on to @timestamp value. Unless I say that the target value is sys\_created\_on again. Can I still use closed\_at the same way? Why was the error only targeted on closed\_at? I still need to covert it. Will it provide me the same error of I use a separate date filter for each column with a target? @Badger ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 30, 2019, 5:30pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/8 "2019-12-30T17:30:29Z")
``` date { match => ["sys_created_on","yyyy-MM-dd HH:mm:ss", "closed_at","yyyy-MM-dd HH:mm:ss", "opened_at","yyyy-MM-dd HH:mm:ss", "resolved_at","yyyy-MM-dd HH:mm:ss", "sys_updated_on","yyyy-MM-dd HH:mm:ss"] } ``` As I said, the date filter takes the value of [sys\_created\_on] and tries to match it against each of the rest of the array entries as a date pattern. During initialization it will build a parser for each of the patterns. It will succeed for "yyyy-MM-dd HH:mm:ss", but then it tries to interpret "closed\_at" as a date pattern, not a field value, and it throws an exception because it is unable to do so. You need something like ``` date { match => ["sys_created_on", "yyyy-MM-dd HH:mm:ss"] target => "sys_created_on" } date { match => ["closed_at", "yyyy-MM-dd HH:mm:ss"] target => "closed_at" } ``` etc. ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 31, 2019, 4:18am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/9 "2019-12-31T04:18:56Z")
Hi @Badger, It still doesn't seem to work out for me. I did try the below, > ``` > date { match => ["sys_created_on", "yyyy-MM-dd HH:mm:ss"] target => "sys_created_on" } > date { match => ["sys_updated_on", "yyyy-MM-dd HH:mm:ss"] target => "sys_updated_on" } > date { match => ["opened_at", "yyyy-MM-dd HH:mm:ss"] target => "opened_at" } > date { match => ["closed_at", "yyyy-MM-dd HH:mm:ss"] target => "closed_at" } > date { match => ["resolved_at", "yyyy-MM-dd HH:mm:ss"] target => "resolved_at" } > > ``` And i still see my columns as a string in Kibana. ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/1/91012a55ee3ef6cb29cfbe7fba27d9f2955a2e3f.png) Note: I also tried`result.`closed\_at, etc, still no result. ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 31, 2019, 9:54am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/10 "2019-12-31T09:54:24Z")
update: i tried to give the target value to a different name, like > date { match =\> ["sys\_created\_on", "yyyy-MM-dd HH:mm:ss"] target =\> "created\_on" } and I see that this created\_on column isn't even created in my ES. Is there a problem with the date filter configuration? Any suggestion/help is appreciated 🙂 Thanks! Katara. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 31, 2019, 3:11pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/11 "2019-12-31T15:11:27Z")
Looking at your xml filter, you have the target option set, so the sys\_created\_on field should not even exist, it would be [Incident][sys\_created\_on] ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 31, 2019, 3:29pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/12 "2019-12-31T15:29:24Z")
@Badger, I'm not using an xml filter anymore. The input was actually json so I'm using > filter > { > json {source =\> "result" } > split{ field =\> ["result"] } > Date {.....} > } However, I can try `[result][sys_created_on]`, based on how you are trying to map it. Is that right? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 31, 2019, 3:31pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/13 "2019-12-31T15:31:51Z")
> [@katara](#): > > date { match =\> ["sys\_created\_on", "yyyy-MM-dd HH:mm:ss"] target =\> "created\_on" } If this does not create [created\_on] and does not add a \_dateparsefailure tag that indicates that [sys\_created\_on] does not exist. I cannot see your data, so I do not know what fields exist. ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [December 31, 2019, 3:59pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/14 "2019-12-31T15:59:23Z")
@Badger, I do get the \_dateparsefailure error tag and I don't see the column getting created. Here's is my sample input, > ``` > {"result":[ > { > "made_sla":"true", > "upon_reject":"Cancel all future Tasks", > "sys_created_on":"2019-12-23 05:00:00", > "number":"INC0010275", > "category":"Network"} , > { > "made_sla":"true", > "upon_reject":"Cancel all future Tasks", > "sys_created_on":"2019-12-24 07:00:00", > "number":"INC0010567", > "category":"DB"}]} > > ``` My elasticsearch index shows these columns prefixed with `result.` to all the columns arriving from the json result set. I've just added a few columns to not make it too messy. However `sys_updated_on, opened_at, closed_at, resolved_at` are the existing date columns, just like the date column sys\_created\_on in the input I've attached. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 31, 2019, 6:57pm UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/15 "2019-12-31T18:57:09Z")
OK, so if you are parsing that JSON and then splitting the result I would expect you to end up with [result][sys\_created\_on], which would indeed show up in kibana as result.sys\_created\_on. Try using "[result][sys\_created\_on]" in the date filter. ---
**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\ **Post date:** [January 2, 2020, 5:12am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/16 "2020-01-02T05:12:50Z")
@Badger , I tried > ``` > date { > match => ["[result][sys_created_on]","yyyy-MM-dd HH:mm:ss"] > target => "closed_at" > } > > ``` and it works perfectly. Thanks much! 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 30, 2020, 5:12am UTC](https://discuss.elastic.co/t/datetime-conversion-error-logsatsh/213319/17 "2020-01-30T05:12:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.