# Trouble matching timestamp **URL:** \ **Category:** Logstash\ **Created:** [April 26, 2017, 8:23pm UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768 "2017-04-26T20:23:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![s58smith](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@s58smith](https://discuss.elastic.co/u/s58smith)\ **Post date:** [April 26, 2017, 8:23pm UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768/1 "2017-04-26T20:23:08Z")
I'm having trouble getting the ISO8601 to match timestamp in "Date" filter. Below is my filter and output config. I have tried using the commented out lines for "match" but not working. > filter { > if "CollectorStatus" == [type] { > date { > #match =\> ["system\_date","yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"] > #match =\> ["system\_date","ISO8601"] > #timezone =\> "UTC" > target =\> "@timestamp" > } > } > } > output { > file { path =\> "/tmp/logstash/stdout\_%{+YYYY.MM.dd}" > codec =\> "json\_lines" > flush\_interval =\> 0 > } > } This is what my output looks like. > {"collection\_date":"2017-04-26T19:45:00.000Z","system\_date":"2017-04-26T20:13:00.000Z","ne\_name":"YXCO41\_FELDZ77-01","ne\_model":"Z77","finished":1,"type":"CollectorStatus","collector\_interval":900,"active\_hdl":"19b","tags":["CollectorStatus","Z77","\_dateparsefailure"],"@timestamp":"2017-04-26T20:15:01.777Z","abort":0,"collector\_name":"YXCO41\_FELDZ77-01","@version":"1","rpu\_hostname":"sapl19"} It is adding the tag "\_dateparsefailure". Based on JSON output I don't think my field name of "system\_date" is nested, so it should be working. Any suggestions? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 27, 2017, 5:30am UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768/2 "2017-04-27T05:30:04Z")
Yeah, using the ISO8601 pattern should definitely work. Check the Logstash log for details about the date parsing failure. ---
**Author:** ![s58smith](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@s58smith](https://discuss.elastic.co/u/s58smith)\ **Post date:** [April 27, 2017, 9:25pm UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768/3 "2017-04-27T21:25:28Z")
Is there anything special I need to put into the conf or logstash startup to get more debug output from the date parser? I've tried with the logstash set with log.level=trace, still isn't helping me. I also changed output to codec=\>"rubydebug" and that isn't helping me. Sample line from /var/log/logstash-plain.log > [2017-04-27T16:30:02,512][DEBUG][logstash.pipeline] output received {"event"=\>{"collection\_date"=\>2017-04-27T20:00:00.000Z, "system\_date"=\>2017-04-27T20:28:00.000Z, "ne\_name"=\>"YXCO41\_FELDZ77-01", "ne\_model"=\>"Z77", "finished"=\>1, "type"=\>"CollectorStatus", "collector\_interval"=\>900, "active\_hdl"=\>"19b", "tags"=\>["NetOptimizeCollectorStatus", "cyan", "\_dateparsefailure"], "collection\_status"=\>"Finished", "@timestamp"=\>2017-04-27T20:30:02.502Z, "abort"=\>0, "collector\_name"=\>"YXCO41\_FELDZ77-01", "@version"=\>"1", "rpu\_hostname"=\>"sapl19", "ne\_vendor"=\>"Cyan"}} Sample line in the rubydebug output. > { > "collection\_date" =\> 2017-04-27T20:45:00.000Z, > "system\_date" =\> 2017-04-27T21:12:00.000Z, > "ne\_name" =\> "TRHLPAXTO02", > "ne\_model" =\> "cyan-Z33", > "finished" =\> 1, > "type" =\> "CollectorStatus", > "collector\_interval" =\> 900, > "active\_hdl" =\> "20a", > "tags" =\> [ > [0] "NetOptimizeCollectorStatus", > [1] "cyan", > [2] "\_dateparsefailure" > ], > "collection\_status" =\> "Finished", > "@timestamp" =\> 2017-04-27T21:15:01.796Z, > "abort" =\> 0, > "collector\_name" =\> "TRHLPAXTO02", > "@version" =\> "1", > "rpu\_hostname" =\> "sapl20", > "ne\_vendor" =\> "Cyan" > } I also searched previous post and found the online tester site [https://joda-time-parse-debugger.herokuapp.com/](https://joda-time-parse-debugger.herokuapp.com/) My date seems to work with the pattern just fine. I'm using the JDBC input for logstash, so is maybe the field "system\_date" not getting into the Date parser? I had multiple conf files before but condensed it to just one for trying to solve this. Below is my ONLY conf file for logstash now. (Note I did alter some fields in JDBC since this is public forum.) > # The # character at the beginning of a line indicates a comment. # Use comments to describe your configuration. input { jdbc { jdbc\_connection\_string =\> "jdbc:oracle:thin:@//fwsd01:1531/PROD" jdbc\_driver\_library =\> "/usr/share/logstash/jdbc\_drivers/oracle-jdbc6.jar" jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_password =\> "xxxxxxxx" jdbc\_user =\> "xxxxxxxx" jdbc\_validate\_connection =\> "true" #parameters =\> { "table" =\> "timezone" } schedule =\> "\*/15 \* \* \* \*" #sql\_log\_level =\> "debug" statement\_filepath =\> "/usr/share/logstash/DB\_queries/c\_status.sql" record\_last\_run =\> "false" tags =\> ["valid","NetOptimizeCollectorStatus","cyan"] type =\> "CollectorStatus" } } # The filter part of this file is commented out to indicate that it is optional. filter { if "CollectorStatus" == [type] { date { match =\> ["system\_date","YYYY-MM-dd'T'HH:mm:ss.SSS'Z'","ISO8601"] timezone =\> "Etc/UTC" target =\> "@timestamp" } } if "valid" not in [tags] { drop { } } mutate { remove\_tag =\> ["valid"] } } output { if "CollectorStatus" == [type] { elasticsearch { hosts =\> ["10.112.91.113:9200"] id =\> "collector\_status" index =\> "collector\_status\_%{+YYYY.MM.dd}" } } file { path =\> "/tmp/logstash/stdout\_%{+YYYY.MM.dd}" codec =\> "rubydebug" flush\_interval =\> 0 } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 5:12am UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768/4 "2017-04-28T05:12:21Z")
> Is there anything special I need to put into the conf or logstash startup to get more debug output from the date parser? No, it logs that at the default log level. > "system\_date" =\> 2017-04-27T21:12:00.000Z, Aha! The field isn't a string but already is a timestamp, and that's why the date filter fails. That's arguably a bug (I've filed [Date filter fails to parse timestamps · Issue #95 · logstash-plugins/logstash-filter-date · GitHub](https://github.com/logstash-plugins/logstash-filter-date/issues/95)) but there are a couple of workarounds that you can try: - In your SQL query, typecast the timestamp as a string. - Use a mutate filter's convert option to typecast the field to a string prior to the date filter. - Use a mutate filter to copy the timestamp into `@timestamp` and overwrite the existing value (use the `replace` option). ---
**Author:** ![s58smith](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@s58smith](https://discuss.elastic.co/u/s58smith)\ **Post date:** [April 28, 2017, 7:58pm UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768/5 "2017-04-28T19:58:35Z")
Thanks very much magnus, Based on your recommendations here is my final solution. I put this solution in case anyone else runs into similar issue. I changed my filter (based on my type) to following. Adding a field in mutate created a string variable. I used this in the data match. Then it gets removed if the match worked. Everything worked great. > if "CollectorStatus" == [type] { > mutate { > add\_field =\> {"temp\_ts" =\> "%{system\_date}"} > } > date { > match =\> ["temp\_ts","ISO8601"] > remove\_field =\> ["temp\_ts"] > } > } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 26, 2017, 8:10pm UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768/6 "2017-05-26T20:10:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.