# "\_grokparsefailure" even though the grok pattern matches **URL:** \ **Category:** Logstash\ **Created:** [August 11, 2017, 1:33pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783 "2017-08-11T13:33:05Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Hisushi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hisushi/32/20551_2.png) [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Post date:** [August 11, 2017, 1:33pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/1 "2017-08-11T13:33:06Z")
Hi everyone, I'm encountering the following problem: I've been testing my grok filter including my patterns on both [Grok Debugger](http://grokdebug.herokuapp.com/) and [Grok Constructor](http://grokconstructor.appspot.com/do/match#result) which works fine. Also, when running logstash with option `-t` no error returns. **But** when running logstash, I still receive a \_grokparsefailure in my tags. So my question is: How is that and how can I start debugging (as I'm pretty sure that my filters/patterns are ok)? I'm using this logstash config ``` file { path => "/media/SAP/log/security_audit_log/*" type => "security_audit_log" start_position => "beginning" codec => plain { charset => "ISO-8859-1" } } } filter { if [type] == "security_audit_log" { mutate { gsub => [ replace => "message", " ", "" ] } grok { patterns_dir => "/media/ELK/logstash-5.5.1/config/paterns/*" match => { "message" => "\|%{SAPDATE:date}\|%{TIME:time}\|%{CLIENT:client}?\|%{USER:username}?%{SPACE}?\|%{TERMINAL:terminal}?%{SPACE}?\|%{TCODE:tcode}?%{SPACE}?\|%{PROGRAM:program}?%{SPACE}?\|%{AUDITCLASS:auditclass}%{SPACE}?\|%{SECURITYLEVEL:securitylevel}%{SPACE}?\|%{MESSAGETEXT:messagetext}%{SPACE}?\|" } } mutate { add_field => { "logtimestamp" => "%{date} %{time}" } remove_field => ["date", "time"] } date { match => ["logtimestamp", "dd.MMM.yyyy HH:mm:ss"] timezone => "Europe/Berlin" locale => "en" target => "@timestamp" } } } output { if [type] == "security_audit_log" { elasticsearch { hosts => "localhost:9200" index => "%{type}-%{timestamp}" } } stdout { codec => "rubydebug" } } ``` with these patterns ``` SAPDATE %{MONTHDAY}.%{MONTHNUM}.%{YEAR} CLIENT \d{3} TERMINAL [\w+.\-]+ TCODE [A-Z0-9_]+ PROGRAM [A-Z0-9_/]+ AUDITCLASS (\w+\s?\-?/?){1,3} SECURITYLEVEL (\w+\s?){1,3} ``` And here you've got some log lines (anonymized) ``` |Date |Time |Cl.|User |Terminal |TCode |Program |Auditclass |Security Level |AuditLog-Messagetext | |20.07.2017|08:01:37| | | | | |System-Ereignisse |Hoch |Applikationsserver gestartet | |24.07.2017|11:17:05|000|SOMEUSER |SOMETERMINAL |SM19 |SAPMSM19 |System-Ereignisse |Hoch |Audit Konfiguration geändert | |24.07.2017|11:17:05|000|SOMEUSER |SOMETERMINAL |SM19 |SAPMSM19 |System-Ereignisse |Hoch |Audit: Slot 1: Klasse 191, Gewicht 5, User * , Mandant 000, | |24.07.2017|11:17:05|000|SOMEUSER |SOMETERMINAL |SM19 |SAPMSM19 |System-Ereignisse |Hoch |Audit Konfiguration geändert | |24.07.2017|11:17:05|000|SOMEUSER |SOMETERMINAL |SM19 |SAPMSM19 |System-Ereignisse |Hoch |Audit: Slot 2 : inaktiv | |24.07.2017|11:19:19|000|SOMEUSER | | |RSBTCRTE |Dialoganmeldung |Mittel |Login erfolgreich (Typ=B, Methode=A ) | |24.07.2017|11:24:14|000|SOMEUSER | | |RSBTCRTE |Dialoganmeldung |Mittel |Login erfolgreich (Typ=B, Methode=A ) | ``` ---
**Author:** ![CDR](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@CDR](https://discuss.elastic.co/u/CDR)\ **Post date:** [August 11, 2017, 1:37pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/2 "2017-08-11T13:37:58Z")
Is \_grokparsefailure showing in all of the log messages or only specific ones? ---
**Author:** ![Hisushi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hisushi/32/20551_2.png) [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Post date:** [August 11, 2017, 1:41pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/3 "2017-08-11T13:41:58Z")
Hi @CDR, it's present in evey single log message. Without exception. ---
**Author:** ![CDR](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@CDR](https://discuss.elastic.co/u/CDR)\ **Post date:** [August 11, 2017, 1:46pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/4 "2017-08-11T13:46:26Z")
The question mark after the **%{SPACE}** is what first sticks out to me. The field **%{SPACE}?** means that there will be zero or one matches of SPACE. In your example logs it seems that there are more than a single space in those fields. Perhaps a better quantifier would be the **\***. So it would be **%{SPACE}\***. This would mean "there is zero or more spaces until the |" ---
**Author:** ![Hisushi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hisushi/32/20551_2.png) [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Post date:** [August 11, 2017, 2:27pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/5 "2017-08-11T14:27:13Z")
I applied your correction but atm it looks like the behavior/\_grokparsefailure did not change. Unfortunately, I lost the connection to the machine that's running my ELK stack, so I'll have a look at the output file later. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 11, 2017, 6:06pm UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/6 "2017-08-11T18:06:31Z")
Any particular reason you're using a grok filter instead of a csv filter? To debug grok problems start simple, e.g. with `^\|%{SAPDATE:date}`. Does that work? If yes, add the next token. ---
**Author:** ![Hisushi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hisushi/32/20551_2.png) [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Post date:** [August 14, 2017, 6:07am UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/7 "2017-08-14T06:07:27Z")
Hi @magnusbaeck, thanks for your reply. Even starting with `^\|%{SAPDATE:date}` gives me a `_grokparsefailure`. I also commented the `mutate` and `date` filter out. To be honest, I wasn't sure whether the `csv` filter would be applicable for my case as I thought it would be better if the fields match a specific pattern instead of looking for columns. Best regards, Hisushi ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 14, 2017, 6:36am UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/8 "2017-08-14T06:36:37Z")
This works for me: ```nohighlight $ cat test.config input { stdin { } } output { stdout { codec => rubydebug } } filter { grok { match => ["message", "^\|%{MONTHDAY}.%{MONTHNUM}.%{YEAR}"] } } $ echo '|20.07.2017' | /opt/logstash/bin/logstash -f test.config Settings: Default pipeline workers: 8 Pipeline main started { "message" => "|20.07.2017", "@version" => "1", "@timestamp" => "2017-08-14T06:36:13.724Z", "host" => "lnxolofon" } Pipeline main has been shutdown stopping pipeline {:id=>"main"} ``` ---
**Author:** ![Hisushi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hisushi/32/20551_2.png) [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Post date:** [August 14, 2017, 7:41am UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/9 "2017-08-14T07:41:00Z")
I guess I figured out the root cause of this problem: When commenting out my `patterns_dir` line, everything works fine, no `_grokparsefailure` when not expected. Some background information: First, I encountered the issue that my patterns have not been "added" when they were located in the path as given in `patterns_dir`. After having a look at the output with the `--debug` option it seemed like logstash was looking for patterns in `/media/ELK/logstash-5.5.1/patterns` while my custom path wasn't given anywhere. So I moved my patterns to the path as expected by Logstash. (and yes, the pattern files are identical) Thank you all for your support! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 11, 2017, 7:41am UTC](https://discuss.elastic.co/t/-grokparsefailure-even-though-the-grok-pattern-matches/96783/10 "2017-09-11T07:41:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.