# Yet another date parse problem (noob)

**URL:** <https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986>\
**Category:** Logstash\
**Created:** [April 13, 2018, 1:39pm UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986 "2018-04-13T13:39:57Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![robs](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@robs](https://discuss.elastic.co/u/robs)\
**Post date:** [April 13, 2018, 1:39pm UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/1 "2018-04-13T13:39:57Z")

</div>

Hi, I want to send log lines to elastic with logstash. I'm burning hours and can't get it right.

My log lines all look like:

yyyymmdd hh:mm:ss ev:1 rn:3   
(like : 20170312 14:03:55 ev:1 rn:5 etc. etc.)

In patterns\_dir I defined the file waxtimestamp with:  
WAXTIMESTAMP %{YEAR}%{MONTHNUM}%{MONTHDAY} %{TIME}

In my filter.conf I have:

input {  
file {  
path =\> ["/var/log/wax/main.log"]  
type =\> "wax"  
}  
}

filter {  
grok {  
patterns\_dir =\> ["./patterns"]  
match =\> { "message" =\> "%{WAXTIMESTAMP:datetime} %{GREEDYDATA:message}" }  
}  
date {  
match =\> { "datetime" =\> "yyyyMMdd HH:mm:ss" }  
target =\> "@timestamp"  
}  
}

output {  
elasticsearch {  
hosts =\> ["elastichost:9200"]  
index =\> "wax-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}

What am I doing wrong?  
In my logstash log I get:

[............]  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@patterns\_dir = ["./patterns"]  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@match = {"message"=\>"%{WAXTIMESTAMP:datetime} %{GREEDYDATA:message}"}  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@id = "5b5d9ebf8eb9cb9e2cd333d92e43b66b76587dab67f16ef87ac31e9906e94a76"  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@enable\_metric = true  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@add\_tag = []  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@remove\_tag = []  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@add\_field = {}  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@remove\_field = []  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@periodic\_flush = false  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@pattern\_definitions = {}  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@patterns\_files\_glob = "\*"  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@break\_on\_match = true  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@named\_captures\_only = true  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@keep\_empty\_captures = false  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@tag\_on\_failure = ["\_grokparsefailure"]  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@timeout\_millis = 30000  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@tag\_on\_timeout = "\_groktimeout"  
[2018-04-13T15:19:13,271][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@overwrite = []  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@match = {"datetime"=\>"yyyyMMdd HH:mm:ss"}  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@target = "@timestamp"  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@id = "3988916c15c511ee38c321a06c6a3513310fb62327e17efd50461c75ce7c1226"  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@enable\_metric = true  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@add\_tag = []  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@remove\_tag = []  
[2018-04-13T15:19:13,272][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@add\_field = {}  
[2018-04-13T15:19:13,273][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@remove\_field = []  
[2018-04-13T15:19:13,273][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@periodic\_flush = false  
[2018-04-13T15:19:13,273][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@tag\_on\_failure = ["\_dateparsefailure"]  
[2018-04-13T15:19:13,273][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"translation missing: en.logstash.agent.configuration.invalid\_plugin\_register",

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 13, 2018, 4:03pm UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/2 "2018-04-13T16:03:52Z")

</div>

grok will successfully match that pattern against your example data, which makes me think the example is not typical. Can you copy and paste an example of a \_grokparsefailure from the JSON tab in Kibana Discover?

---

<div class="post-metadata">

**Author:** ![robs](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@robs](https://discuss.elastic.co/u/robs)\
**Post date:** [April 16, 2018, 9:42am UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/3 "2018-04-16T09:42:28Z")

</div>

Hi Thank you for your prompt response.  
Logstash & elastic are running on a (non-graphics) Linux server. Kibana is running on my laptop.  
Because of the error, nothing is put into elastissearch. The index mentioned above doesn't exist.  
Every single line starts with yyyymmdd hh:mm:ss so every single line fails.  
I can't understand the error-message.  
(message=\>"translation missing: en.logstash.agent.configuration.invalid\_plugin\_register")  
Is my logstash missing something?  
Is there another way to parse these data fields?  
Thank in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2018, 11:50am UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/4 "2018-04-16T11:50:58Z")

</div>

That's not a very friendly error message, but you have

> [@robs](#):
>
> date {  
> match =\> { "datetime" =\> "yyyyMMdd HH:mm:ss" }  
> target =\> "@timestamp"  
> }

match takes an array, not a hash. Change it to be

```auto
match => ["datetime", "yyyyMMdd HH:mm:ss"]

```

---

<div class="post-metadata">

**Author:** ![robs](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@robs](https://discuss.elastic.co/u/robs)\
**Post date:** [April 16, 2018, 12:27pm UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/5 "2018-04-16T12:27:11Z")

</div>

Hi Badger, You nailed it! Thanks a bundle. As you can see, this is all new to me.  
Your help brought me further along this obstacle course. Right up to:

[2018-04-16T14:11:14,397][DEBUG][logstash.filters.grok] Adding pattern {"MONGO\_WORDDASH"=\>"\b[\w-]+\b"}  
[2018-04-16T14:11:14,397][DEBUG][logstash.filters.grok] Adding pattern {"MONGO3\_SEVERITY"=\>"\w"}  
[2018-04-16T14:11:14,397][DEBUG][logstash.filters.grok] Adding pattern {"MONGO3\_COMPONENT"=\>"%{WORD}|-"}  
[2018-04-16T14:11:14,397][DEBUG][logstash.filters.grok] Adding pattern {"MONGO3\_LOG"=\>"%{TIMESTAMP\_ISO8601:timestamp} %{MONGO3\_SEVERITY:severity} %{MONGO3\_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])? %{GREEDYDATA:message}"}  
[2018-04-16T14:11:14,397][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::FilterDelegator:0x6ddd72d3 @metric\_events\_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @id="9cb4743ebb4326504e91a48cf9682dae4306bd3ad235fc864afa6cee19a73388", @klass=LogStash::Filters::Grok, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0x54465efc @metric=#\<LogStash::Instrument::Metric:0x23c84e4 @collector=#\<LogStash::Instrument::Collector:0x3ee50ae5 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x3c5d2c19 @store=#\<Concurrent:🗺0x00000000000fb4 entries=3 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x19353990, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb8 entries=73 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :filters, :"9cb4743ebb4326504e91a48cf9682dae4306bd3ad235fc864afa6cee19a73388", :events]\>, @filter=\<LogStash::Filters::Grok patterns\_dir=\>["./patterns"], match=\>{"message"=\>"%{WAXTIMESTAMP:datetime} %{GREEDYDATA:message}"}, id=\>"9cb4743ebb4326504e91a48cf9682dae4306bd3ad235fc864afa6cee19a73388", enable\_metric=\>true, periodic\_flush=\>false, patterns\_files\_glob=\>"\*", break\_on\_match=\>true, named\_captures\_only=\>true, keep\_empty\_captures=\>false, tag\_on\_failure=\>["\_grokparsefailure"], timeout\_millis=\>30000, tag\_on\_timeout=\>"\_groktimeout"\>\>", :error=\>"pattern %{WAXTIMESTAMP:datetime} not defined", :thread=\>"#\<Thread:0x742df586@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run\>"}  
[2018-04-16T14:11:14,398][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<Grok::PatternError: pattern %{WAXTIMESTAMP:datetime} not defined\>, :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1292:in`loop'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in `compile'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:in`block in register'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:in`block in register'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in`register\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:736:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:362:in`start\_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:289:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:249:in`block in start'"], :thread=\>"#\<Thread:0x742df586@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run\>"}

I would be very obliged if you or anyone can explain that one to me, or failing that, how to get around it.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2018, 12:34pm UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/6 "2018-04-16T12:34:27Z")

</div>

> [@robs](#):
>
> :error=\>"pattern %{WAXTIMESTAMP:datetime} not defined"

Is ./patterns really in the directory where logstash is running? Perhaps try an absolute path.

---

<div class="post-metadata">

**Author:** ![robs](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@robs](https://discuss.elastic.co/u/robs)\
**Post date:** [April 17, 2018, 8:06am UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/7 "2018-04-17T08:06:32Z")

</div>

Well Badger, you nailed it again. My .patterns was relative to the config/settings directory, which I erroneously thought to be correct.  
I see I need to parse error messages more carefully (didn't notice the "not defined" part in all that jazz or I _might_ have had an inkling) and re-study a lot of docs.  
Bottomline: it _works_. Thank you very much.  
κύδος to you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2018, 8:06am UTC](https://discuss.elastic.co/t/yet-another-date-parse-problem-noob/127986/8 "2018-05-15T08:06:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.