# Date FIlter - \_dateparsefailure \[SOLVED\] **URL:** \ **Category:** Logstash\ **Created:** [November 2, 2016, 9:51am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692 "2016-11-02T09:51:18Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 9:51am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/1 "2016-11-02T09:51:18Z")
Hi, I'm new with the date filter. I set up this configuration : ``` filter { if "mail" in [type] and "maillog" in [source] { grok { match => ["message","%{CISCOTIMESTAMP:mail_date} %{DATA:server_name} %{DATA:service}/%{DATA:process}\[%{DATA:pid}\]:", "message","%{CISCOTIMESTAMP:mail_date} %{DATA:server_name} %{DATA:service}\[%{DATA:pid}\]:"] break_on_match => true } date { match => ["mail_date", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } } } ``` To match this timestamp : `Nov 2 10:40:49` I see a `_dateparsefailure` in the tags field, i don't know why this tag appeared and how to fix the problem. ---
**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\ **Post date:** [November 2, 2016, 9:55am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/2 "2016-11-02T09:55:18Z")
It seems to work fine with that date pattern: ```auto % echo "Nov 2 10:40:49" | bin/logstash -e 'input { stdin {} } filter { date { match => ["message", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } }' { "@timestamp" => 2016-11-02T10:40:49.000Z, "@version" => "1", "host" => "Joaos-MBP-5.lan", "message" => "Nov 2 10:40:49" } ``` Are those events also populated with the \_grokparsefailure? or just the \_dateparsefailure? ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 10:02am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/3 "2016-11-02T10:02:01Z")
Just the \_dateparsefailure I agree, it seems to work, i just have this tags... Do you think a `remove_tag` will be a good option ? ---
**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\ **Post date:** [November 2, 2016, 10:04am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/4 "2016-11-02T10:04:21Z")
if that tag is applied then the date filter is failing..this should be investigated. could you be running another configuration file alongside this one? ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 10:09am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/5 "2016-11-02T10:09:01Z")
Yes, i have other configurations files alongside this one ---
**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\ **Post date:** [November 2, 2016, 10:10am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/6 "2016-11-02T10:10:09Z")
it could happen that the events are also passing to the other configuration file which also have date filters. If so you need to isolate the flows using tags/fields and conditionals ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 10:12am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/7 "2016-11-02T10:12:23Z")
I don't have any other date filter for the moment. And i already use conditionnals to root the log to the correct filter. ---
**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\ **Post date:** [November 2, 2016, 10:30am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/8 "2016-11-02T10:30:04Z")
can you post the full configuration and some logging at debug level? ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 10:41am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/9 "2016-11-02T10:41:00Z")
here is my main conf file : ``` input { redis { host => "127.0.0.1" port => 6379 data_type => "list" type => "redis-input" key => "logstash" } beats { port => 5045 type => "filebeat-input" congestion_threshold => 99999999 } } output { if "_grokparsefailure" not in [tags] { if "tango" in [tags] { elasticsearch { hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"] index => "logstash-tango-%{+YYYY.MM.dd}" } } if "nginx" in [tags] { elasticsearch { hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"] index => "logstash-isg-%{+YYYY.MM.dd}" } } if "scarlette" in [tags] { elasticsearch { hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"] index => "logstash-scarlette-%{+YYYY.MM.dd}" } } if "serveur_owncloud" in [tags] { elasticsearch { hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"] index => "logstash-owncloud-%{+YYYY.MM.dd}" } } if "brouette" in [tags] or "poussette" in [tags] { elasticsearch { hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"] index => "logstash-mta-%{+YYYY.MM.dd}" } } if "serveur_proxy" in [tags] or "serveur_dns" in [tags] { elasticsearch { hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"] index => "logstash-proxydns-%{+YYYY.MM.dd}" } } } } ``` Here is my filter with the date filter : ``` filter { if "mail" in [type] and "maillog" in [source] { grok { match => ["message","%{CISCOTIMESTAMP:mail_date} %{DATA:server_name} %{DATA:service}/%{DATA:process}\[%{DATA:pid}\]:", "message","%{CISCOTIMESTAMP:mail_date} %{DATA:server_name} %{DATA:service}\[%{DATA:pid}\]:"] break_on_match => true } date { match => ["mail_date", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] timezone => "Europe/Paris" } } } ``` ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 10:45am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/10 "2016-11-02T10:45:35Z")
My log who match with the date filter, aslo match with another filter. I can't show you this filter here because it contain to many characters... ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [November 2, 2016, 10:50am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/11 "2016-11-02T10:50:39Z")
Try a different order of the values in the date match list. e.g. ```auto ["mail_date", "MMM dd HH:mm:ss", "MMM d HH:mm:ss"] ``` Do you get `_dateparsefailure` it this case? ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 10:57am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/12 "2016-11-02T10:57:12Z")
Yes i also have `_dateparsefailure`in this case. Maybe the error is just because the date field doesn't match with the pattern `MMM dd HH:mm:ss` But when the date will be `Nov 11 10:40:49` instead of `Nov 2 10:40:49`, the date field will continue to match with the pattern `MMM d HH:mm:ss` ? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [November 2, 2016, 10:57am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/13 "2016-11-02T10:57:41Z")
Also according to [RFC 3164 The BSD syslog Protocol](https://tools.ietf.org/html/rfc3164#section-4.1.2) a singular day (1-9) should have a space in front e.g. `Nov 2 10:10:10`. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [November 2, 2016, 10:58am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/14 "2016-11-02T10:58:45Z")
Your pattern should be `MMMd HH:mm:ss` ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 11:02am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/15 "2016-11-02T11:02:37Z")
This problem of double space fix my problem thanks ! ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [November 2, 2016, 11:06am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/16 "2016-11-02T11:06:56Z")
@Clement_Ros The date filter will try to match either pattern. Unfortunately it uses a slow mechanism to "see" the failure from the earlier pattern before trying the later patterns. This means that you should put the most likeliest pattern first, and because there are more days from 10 to the end of a month - you will get better performance checking `MMMddHH:mm:ss` first when the syslog message is from days 10 through 28|19|30|31 of any month. ---
**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Post date:** [November 2, 2016, 11:54am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/17 "2016-11-02T11:54:32Z")
Thanks for the help, i understood 😀 ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [November 2, 2016, 11:56am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/18 "2016-11-02T11:56:21Z")
You are welcome. ---
**Author:** ![Bharath\_Raju](https://avatars.discourse-cdn.com/v4/letter/b/7ea924/32.png) [@Bharath\_Raju](https://discuss.elastic.co/u/Bharath_Raju)\ **Post date:** [June 22, 2017, 12:08am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/19 "2017-06-22T00:08:03Z")
I have a similar issue. Help is much appreciated - new here:grin: Ful config is - ``` input { stdin {} } filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:@timestamp}" , } } date { match => { "@timestamp" => "yyyy MMM dd HH:mm:ss"} } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["localhost:9200"] index => "test1-%{+YYYY.MM.dd}" } } ``` And the Input is - ``` 2017 Jun 13 01:56:51 ``` ---
**Author:** ![Bharath\_Raju](https://avatars.discourse-cdn.com/v4/letter/b/7ea924/32.png) [@Bharath\_Raju](https://discuss.elastic.co/u/Bharath_Raju)\ **Post date:** [June 22, 2017, 12:09am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692/20 "2017-06-22T00:09:38Z")
The logs are as below - ``` > [2017-06-22T05:31:52,846][DEBUG][logstash.runner] --------------- Logstash Settings ------------------- > [2017-06-22T05:31:52,881][DEBUG][logstash.agent] Agent: Configuring metric collection > [2017-06-22T05:31:52,883][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120} > [2017-06-22T05:31:52,899][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120} > [2017-06-22T05:31:52,930][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120} > [2017-06-22T05:31:52,946][DEBUG][logstash.agent] Reading config file {:config_file=>"C:/elastic/logstash/bin/test1.conf"} > [2017-06-22T05:31:53,084][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"stdin", :type=>"input", :class=>LogStash::Inputs::Stdin} > [2017-06-22T05:31:53,115][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"line", :type=>"codec", :class=>LogStash::Codecs::Line} > [2017-06-22T05:31:53,184][DEBUG][logstash.inputs.stdin] config LogStash::Inputs::Stdin/@add_field = {} > [2017-06-22T05:31:53,221][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"grok", :type=>"filter", :class=>LogStash::Filters::Grok} > [2017-06-22T05:31:53,230][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@match = {"message"=>"%{TIMESTAMP_ISO8601:@timestamp}"} > [2017-06-22T05:31:53,236][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@id = "a37ca75f76b51ef1203771d08a1b35b3a67fe7b1-2" > [2017-06-22T05:31:53,262][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@periodic_flush = false > [2017-06-22T05:31:53,267][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@patterns_dir = [] > [2017-06-22T05:31:53,275][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@pattern_definitions = {} > [2017-06-22T05:31:53,281][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@patterns_files_glob = "*" > [2017-06-22T05:31:53,302][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@tag_on_failure = ["_grokparsefailure"] > [2017-06-22T05:31:53,306][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@timeout_millis = 30000 > [2017-06-22T05:31:53,310][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@tag_on_timeout = "_groktimeout" > [2017-06-22T05:31:53,314][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@overwrite = [] > [2017-06-22T05:31:53,324][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"date", :type=>"filter", :class=>LogStash::Filters::Date} > [2017-06-22T05:31:53,340][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@match = {"@timestamp"=>"yyyy MMM dd HH:mm:ss"} > [2017-06-22T05:31:53,340][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@id = "a37ca75f76b51ef1203771d08a1b35b3a67fe7b1-3" > [2017-06-22T05:31:53,355][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@enable_metric = true > [2017-06-22T05:31:53,355][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@add_tag = [] > [2017-06-22T05:31:53,355][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@remove_tag = [] > [2017-06-22T05:31:53,355][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@add_field = {} > [2017-06-22T05:31:53,355][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@remove_field = [] > [2017-06-22T05:31:53,355][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@periodic_flush = false > [2017-06-22T05:31:53,371][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@target = "@timestamp" > [2017-06-22T05:31:53,371][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@tag_on_failure = ["_dateparsefailure"] > [2017-06-22T05:31:53,387][ERROR][logstash.agent] Cannot create pipeline {:reason=>"translation missing: en.logstash.agent.configuration.invalid_plugin_register", :backtrace=>["C:/elastic/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-date-3.1.5/lib/logstash/filters/date.rb:162:in `initialize'", "C:/elastic/logstash/logstash-core/lib/logstash/filter_delegator.rb:21:in `initialize'", "C:/elastic/logstash/logstash-core/lib/logstash/pipeline.rb:96:in `plugin'", "(eval):37:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "C:/elastic/logstash/logstash-core/lib/logstash/pipeline.rb:63:in `initialize'", "C:/elastic/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "C:/elastic/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "C:/elastic/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "C:/elastic/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "C:/elastic/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "C:/elastic/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "C:/elastic/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "C:\\elastic\\logstash\\lib\\bootstrap\\environment.rb:71:in `(root)'"]} > [2017-06-22T05:31:53,424][DEBUG][logstash.agent] starting agent > [2017-06-22T05:31:53,440][DEBUG][logstash.agent] Starting puma > [2017-06-22T05:31:53,440][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping > [2017-06-22T05:31:53,440][DEBUG][logstash.agent] Trying to start WebServer {:port=>9600} > [2017-06-22T05:31:53,440][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping > [2017-06-22T05:31:53,440][DEBUG][logstash.api.service] [api-service] start > [2017-06-22T05:31:53,456][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping ``` [Next page](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692.md?page=2)