# Trying to parse this datefield **URL:** \ **Category:** Logstash\ **Created:** [June 3, 2018, 6:25pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331 "2018-06-03T18:25:29Z")\ **Posts on this page:** 19\ **Page:** 2
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 5, 2018, 6:58pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/21 "2018-06-05T18:58:17Z")
Yes, I am using: `curl -X DELETE 'http://localhost:9200/_all'` to clear everything each time. I'm gonna retry. I'll keep you updated on what happens. Thank you ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 5, 2018, 6:58pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/22 "2018-06-05T18:58:46Z")
Makes much more sense, This is gonna make my life easier while debugging, Thanks! I'm gonna give it a shot now ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 5, 2018, 7:04pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/23 "2018-06-05T19:04:37Z")
Starting up logstash is pretty expensive. You may find it easier to do ``` -r --path.settings -f --debug ``` The -r tells it to re-read the config file every time it changes, then you can feed it another line on stdin after the re-load. ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 6, 2018, 6:02pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/24 "2018-06-06T18:02:01Z")
okay. this is officially driving me crazy. I can see that the date is failing to parse in the generated JSON. but there's no related logs in logstash logs. I also tried all different combinations i can think off. Can you guys please take a look? **Here's my filter:** ``` input { stdin { id => "my_plugin_id" } } filter { csv { columns => [ "logdate", "ppoeip", "userip", "protocol", "logserverip", "destinationip", "remove1", "remove2", "sourceport", "destinationport", "description-url" ] separator => ";" remove_field => ["remove1", "remove2"] } date { match => ["logdate", "MMM dd, yyyy HH:mm:ss.SSS ZZZ"] target => "logdate" } } output { stdout { } } ``` **Here's my Input:** `"May 31, 2018 09:35:48.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello" | /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf --debug` **Here's the output:** ``` Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties { "ppoeip" => "172.25.35.1", "logserverip" => "172.25.35.24", "message" => "May 31, 2018 09:35:48.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello", "description-url" => "Client Hello", "tags" => [ [0] "_dateparsefailure" ], "destinationip" => "216.58.201.202", "sourceport" => "8897", "destinationport" => "8897", "protocol" => "SSL", "@timestamp" => 2018-06-06T17:54:27.793Z, "logdate" => "May 31, 2018 09:35:48.979371597 EEST", "@version" => "1", "host" => "Iclik-LOG", "userip" => "10.8.104.200" } ``` **And Finally, Here's the log:** ``` [2018-06-06T20:54:27,459][DEBUG][logstash.filters.csv] config LogStash::Filters::CSV/@convert = {} [2018-06-06T20:54:27,460][DEBUG][logstash.filters.csv] config LogStash::Filters::CSV/@autodetect_column_names = false [2018-06-06T20:54:27,466][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"date", :type=>"filter", :class=>LogStash::Filters::Date} [2018-06-06T20:54:27,479][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@match = ["logdate", "MMM dd, yyyy HH:mm:ss.SSS ZZZ"] [2018-06-06T20:54:27,479][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@target = "logdate" [2018-06-06T20:54:27,479][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@id = "82ee11addf342f61e7d743aadc55cc4207e69cfe-3" [2018-06-06T20:54:27,479][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@enable_metric = true [2018-06-06T20:54:27,479][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@add_tag = [] [2018-06-06T20:54:27,480][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@remove_tag = [] [2018-06-06T20:54:27,480][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@add_field = {} [2018-06-06T20:54:27,480][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@remove_field = [] [2018-06-06T20:54:27,480][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@tag_on_failure = ["_dateparsefailure"] [2018-06-06T20:54:27,492][DEBUG][org.logstash.filters.DateFilter] Date filter with format=MMM dd, yyyy HH:mm:ss.SSS ZZZ, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser [2018-06-06T20:54:27,508][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"stdout", :type=>"output", :class=>LogStash::Outputs::Stdout} [2018-06-06T20:54:27,514][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"rubydebug", :type=>"codec", :class=>LogStash::Codecs::RubyDebug} [2018-06-06T20:54:27,517][DEBUG][logstash.codecs.rubydebug] config LogStash::Codecs::RubyDebug/@id = "rubydebug_ed456ced-9011-4c54-8989-4fce52c275e2" [2018-06-06T20:54:27,643][DEBUG][logstash.filters.csv] CSV parsing options {:col_sep=>";", :quote_char=>"\""} [2018-06-06T20:54:27,645][INFO][logstash.pipeline] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000} [2018-06-06T20:54:27,816][DEBUG][logstash.pipeline] filter received {"event"=>{"@version"=>"1", "host"=>"Iclik-LOG", "@timestamp"=>2018-06-06T17:54:27.793Z, "message"=>"May 31, 2018 09:35:48.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello"}} [2018-06-06T20:54:27,817][DEBUG][logstash.filters.csv] Running csv filter {:event=>2018-06-06T17:54:27.793Z Iclik-LOG May 31, 2018 09:35:48.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello} [2018-06-06T20:54:27,858][DEBUG][logstash.filters.csv] filters/LogStash::Filters::CSV: removing field {:field=>"remove1"} [2018-06-06T20:54:27,859][DEBUG][logstash.filters.csv] filters/LogStash::Filters::CSV: removing field {:field=>"remove2"} [2018-06-06T20:54:27,859][DEBUG][logstash.filters.csv] Event after csv filter {:event=>2018-06-06T17:54:27.793Z Iclik-LOG May 31, 2018 09:35:48.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello} [2018-06-06T20:54:27,893][DEBUG][logstash.pipeline] output received {"event"=>{"ppoeip"=>"172.25.35.1", "logserverip"=>"172.25.35.24", "message"=>"May 31, 2018 09:35:48.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello", "description-url"=>"Client Hello", "tags"=>["_dateparsefailure"], "destinationip"=>"216.58.201.202", "sourceport"=>"8897", "destinationport"=>"8897", "protocol"=>"SSL", "@timestamp"=>2018-06-06T17:54:27.793Z, "logdate"=>"May 31, 2018 09:35:48.979371597 EEST", "@version"=>"1", "host"=>"Iclik-LOG", "userip"=>"10.8.104.200"}} ``` Now I Know that According to the docs, EEST is not a supported timezone. But what Am I supposed to do in this case? Thanks again! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2018, 6:07pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/25 "2018-06-06T18:07:03Z")
> [@TestCandidate](#): > > Now I Know that According to the docs, EEST is not a supported timezone. But what Am I supposed to do in this case? Transform it into something that is has the same offset from UTC, but is supported. According to Wikipedia Moscow time is the same as EEST. ``` mutate { gsub => ["logdate", "EEST", "Europe/Moscow"] } ``` ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 6, 2018, 6:42pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/26 "2018-06-06T18:42:55Z")
AWESOME. That solved the problem. The date is parsed correctly. However, unlike the original timestamp, logdate still appears as string and not as a datefield for some reason. is there any missing step to add? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2018, 7:00pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/27 "2018-06-06T19:00:37Z")
> [@TestCandidate](#): > > However, unlike the original timestamp, logdate still appears as string and not as a datefield for some reason. is there any missing step to add? You mean in Kibana? Did you start with an empty index and refresh the fields of the index pattern? ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 6, 2018, 7:06pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/28 "2018-06-06T19:06:06Z")
yes in Kibana. I did the following: ``` curl -X DELETE 'http://localhost:9200/_all' rm /var/lib/filebeat/registry ``` And then started logstash then filebeat ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2018, 7:15pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/29 "2018-06-06T19:15:43Z")
Go to Management in the left nav, then Index Patterns, then find your pattern and do a refresh at the top right. ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 6, 2018, 8:11pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/30 "2018-06-06T20:11:02Z")
The whole index was deleted and i added the filebeat-\* indexes. but anyway, I tried to refresh the index patterns. Unfortunately, it says it's a string 😕 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2018, 8:20pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/31 "2018-06-06T20:20:38Z")
What does output { stdout { codec =\> rubydebug } } produce? ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 7, 2018, 5:03pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/32 "2018-06-07T17:03:32Z")
> [@Badger](#): > > output { stdout { codec =\> rubydebug } } From the logs? Here's what I've got after putting your output setting: `[2018-06-07T20:00:54,051][DEBUG][logstash.pipeline] output received {"event"=>{"ppoeip"=>"172.25.35.1", "logserverip"=>"172.25.35.24", "offset"=>127, "input_type"=>"log", "source"=>"/var/capture/faridtest/farid.csv", "message"=>"May 31, 2018 09:35:45.979371597 EEST;172.25.35.1;10.8.104.200;SSL;172.25.35.24;216.58.201.202;27786;443;8897;8897;Client Hello", "type"=>"iclick-tcplog", "tags"=>["beats_input_codec_plain_applied"], "description-url"=>"Client Hello", "destinationip"=>"216.58.201.202", "sourceport"=>"8897", "destinationport"=>"8897", "protocol"=>"SSL", "@timestamp"=>2018-06-07T17:00:45.711Z, "logdate"=>2018-05-31T06:35:45.979Z, "@version"=>"1", "beat"=>{"name"=>"Iclik-LOG", "hostname"=>"Iclik-LOG", "version"=>"5.6.9"}, "host"=>"Iclik-LOG", "userip"=>"10.8.104.200"}}` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 7, 2018, 5:32pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/33 "2018-06-07T17:32:24Z")
> [@TestCandidate](#): > > "logdate"=\>2018-05-31T06:35:45.979Z OK, there are no quotes around the value, so at that point it is a logstash timestamp, not a string. Do you have a template that matches that index? If not, I suspect you are not actually deleting the index. (Maybe you are deleting all the docs, but leaving metadata behind?) Can you delete the index by name? If you run something like this. It shows up as a date, right? ```auto input { generator { message => '2018/05/11 12:34:56' count => 1 } } filter { date { match => ["message", "yyyy/MM/dd HH:mm:ss"] target => "foo" remove_field => ["host", "message"] } } output { elasticsearch { index => "deleteme" hosts => ["localhost"] } } ``` The clean up with ``` DELETE /deleteme ``` ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 7, 2018, 6:23pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/34 "2018-06-07T18:23:21Z")
Doesn't this mean that all indexes are deleted? This appears every time I run: > curl -X DELETE '[http://localhost:9200/\_all](http://localhost:9200/_all) ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/c/e/ce3df5cf3724cd3b78b8c1a4957f134f9098aa7a.JPG) **PS: foo did appear as date and not as string** ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 7, 2018, 7:06pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/35 "2018-06-07T19:06:31Z")
Could another instance be loading data that is string? If you go to discover and query for _exists_: logdate and then Add logdate do you see what you expect? I ran it in as a string first, then as a date (which gets converted to string since the type is already set). And I see this ![AA1](https://us1.discourse-cdn.com/elastic/original/3X/7/d/7daa17d2d9067166f749dec0b8436cf99f7869fb.png) It might be worth starting a new question taking filebeat out of the question and using a minimal configuration like ```auto input { generator { message => 'May 31, 2018 09:35:45.979371597 EEST' count => 1 } } filter { mutate { gsub => ["message", "EEST", "Europe/Moscow"] } date { match => ["message", "MMM dd, yyyy HH:mm:ss.SSSSSSSSS ZZZ"] target => "logdate" } } output { stdout { codec => rubydebug } } output { elasticsearch {} } ``` but I think you will just get told that you either didn't delete the index, or didn't refresh the fields. ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 8, 2018, 8:12pm UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/36 "2018-06-08T20:12:17Z")
I'll give it a fresh start and get back with the result. since logstash is parsing correctly, as you said, it must be something with the index. I'll let you know to keep track if anyone had any similar problem in the future. Thanks again! ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 9, 2018, 7:47am UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/37 "2018-06-09T07:47:20Z")
No Matter What i do, it's always a string. \* Sighs\* Can you please take a look at my output configuration? Am I missing something here? ``` output { elasticsearch { hosts => ["localhost:9200"] # sniffing => true # manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" } } ``` ---
**Author:** ![TestCandidate](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Post date:** [June 9, 2018, 9:30am UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/38 "2018-06-09T09:30:23Z")
Finally, It worked after I Upgraded the whole stack from 5.x to 6.x. not sure if it was the reason, or it magically solved the problem. but, it worked. Thank you all for your awesome support ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 7, 2018, 9:30am UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331/39 "2018-07-07T09:30:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331.md?page=1)