# \_grokparsefailure but grok debugger looks good **URL:** \ **Category:** Logstash\ **Created:** [July 21, 2021, 1:06pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269 "2021-07-21T13:06:08Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 21, 2021, 1:06pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/1 "2021-07-21T13:06:08Z")
Hello everyone, I need some help with my grok pattern, because Logstash is not able to parse it. Kibana and other grok debuggers are able to parse the logs. Sample Log: ```auto Mar 10 00:59:50\t1.1.1.1\ttest@static\t0/0/0/0000\tunknown\tstart\ttask_id=67511\ttimezone=met\tservice=ppp ``` Grok Pattern ```auto %{MONTH:month}\s*%{MONTHDAY:day}\s*%{TIME:time}\\t%{IPV4:IGP}\\t%{PPPOEUSER:pppoeuser}\\t%{INTERFACE:Interface}\\t%{GREEDYDATA:tac_message} ``` Custom Patterns: ```auto PPPOE [.a-zA-Z0-9_-]+ STATIC [.a-zA-Z0-9_-]+ PPPOEUSER %{PPPOE}@%{STATIC} INTERFACE [\d/\d/\d/\d]+ ``` Config: ```auto input { beats { port => 5050 ssl => false } } filter { grok { patterns_dir => ["/usr/share/logstash/patterns"] match => { "message" => "%{MONTH:month}\s*%{MONTHDAY:day}\s*%{TIME:time}\\t%{IPV4:IGP}\\t%{PPPOEUSER:pppoeuser}\\t%{INTERFACE:Interface}\\t%{GREEDYDATA:tac_message}" } } } output { elasticsearch { hosts => [localhost:9200"] index => "index-%{+YYYY.MM.dd}" } # debug stdout { codec => rubydebug } } ``` Thank you! ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 21, 2021, 1:25pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/2 "2021-07-21T13:25:55Z")
Hi, Can you show us the result of > [@baumi](#): > > `stdout { codec => rubydebug }` Cad. ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 21, 2021, 1:33pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/3 "2021-07-21T13:33:21Z")
Hi, of course ```auto { "input" => { "type" => "log" }, "@version" => "1", "@timestamp" => 2021-07-21T12:27:08.769Z, "log" => { "file" => { "path" => "/var/log/log.log" }, "offset" => 10403402 }, "ecs" => { "version" => "1.8.0" }, "agent" => { "id" => "ddc8951e-5c80-4613-a670-91f0f60b50d9", "version" => "7.13.4", "ephemeral_id" => "9e1a57c6-62db-4bb5-a2a8-579a3fc75044", "name" => "system02", "type" => "filebeat", "hostname" => "system02" }, "message" => "Mar 10 00:59:50\t1.1.1.1\ttest@static\t0/0/0/0000\tunknown\tstart\ttask_id=67511\ttimezone=met\tservice=ppp", "tags" => [ [0] "beats_input_codec_plain_applied", [1] "_grokparsefailure" ], "host" => { "id" => "90b06be9d8564ff694298df08bc29f26", "mac" => [ [0] "00:00:00:00:00:00" ], "hostname" => "system02", "containerized" => false, "name" => "system02", "architecture" => "x86_64", "ip" => [ [0] "1.1.1.2", ], "os" => { "codename" => "Core", "version" => "8 (Core)", "name" => "CentOS Linux", "kernel" => "4.18.0-193.14.2.el8_2.x86_64", "type" => "linux", "family" => "redhat", "platform" => "centos" } } ``` Best regards ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 21, 2021, 1:55pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/4 "2021-07-21T13:55:56Z")
I don't see any error in the grok configuration. Are you sure the conf file you give to us is the file you are running in logstash ? Because this line have to give you an error. > [@baumi](#): > > `hosts => [localhost:9200"]` ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 21, 2021, 2:01pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/5 "2021-07-21T14:01:37Z")
This is the correct line. ```auto hosts => ["master1:9200", "master2:9200"] ``` Overall the configuration should be ok, because logstash does not fail when it starts and I can see the logs in Kibana. ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 21, 2021, 2:41pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/6 "2021-07-21T14:41:41Z")
Maybe it's because of INTERFACE `\d` is for found digit, so INTERFACE search multiple group of 4 digit seperated by slash. In your case, INTERFACE have to take 3digit and un number all seperated by a slash. So INTERFACE need to be declared like this `INTERFACE \d/\d/\d/[\d]+` ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 21, 2021, 2:47pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/7 "2021-07-21T14:47:53Z")
> [@Cad](#): > > INTERFACE \d/\d/\d/[\d]+ I tried the pattern but it did not work. Still grokparsefailure ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 21, 2021, 3:15pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/8 "2021-07-21T15:15:49Z")
So, i think, the better way to find who's giving an error is to start with DATA patterns `%{DATA:date}\\t%{DATA:IGP}\\t%{DATA:pppoeuser}\\t%{DATA:Interface}\\t%{GREEDYDATA:tac_message}` And adding one by one the patterns you want until logstash show one error. I made a mistake in my last post, about your first pattern of INTERFACE. I tell, > [@Cad](#): > > INTERFACE search multiple group of 4 digit seperated by slash. It's false, your pattern literaly tell "i search a digit or a slahs or a digit or a slahs..." 4 times. I still think the pattern i recommand you fit more your data. ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 22, 2021, 6:12am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/9 "2021-07-22T06:12:18Z")
Logstash is already showing \_grokparsefailure when using only DATA pattern for date. Just going by following pattern will not give an error. So the date format has to be the problem. ```auto %{GREEDYDATA:tac_message} ``` I already recognized that there are two spaces between month and day. That's why I used ```auto %{MONTH:month}\s*%{MONTHDAY:day} ``` ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 22, 2021, 7:14am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/10 "2021-07-22T07:14:41Z")
A pattern already exist for this date format it is the [SYSLOGTIMESTAMP](https://github.com/hpcugent/logstash-patterns/blob/aaede7a2e508c1a37816bff9d9824772f1eed78d/files/grok-patterns#L83). ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 22, 2021, 7:54am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/11 "2021-07-22T07:54:48Z")
Still \_grokparsefailure. I added `SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}` to my patterns file and edited my grok filter. ```auto match => { "message" => "%{SYSLOGTIMESTAMP:tac_timestamp}\\t%{GREEDYDATA:tac_message}" } ``` ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 22, 2021, 8:04am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/12 "2021-07-22T08:04:46Z")
Have you tried to use another format for the tab ? Like `\\[t]` or `[\\][t]` ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 22, 2021, 8:58am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/13 "2021-07-22T08:58:09Z")
Yes I already tried that but still grokparsefailure. ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 22, 2021, 10:21am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/14 "2021-07-22T10:21:05Z")
I try to parse your line with my own logstash and it work well. I try this two grok configuration : > [@baumi](#): > > `"%{SYSLOGTIMESTAMP:tac_timestamp}\\t%{GREEDYDATA:tac_message}"` > [@Cad](#): > > %{DATA:date}\t%{DATA:IGP}\t%{DATA:pppoeuser}\t%{DATA:Interface}\t%{GREEDYDATA:tac\_message} Each time, i got the good result without any grokparsefailure tag. What do you do after editing the grok pattern to reload logstash ? ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 22, 2021, 10:30am UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/15 "2021-07-22T10:30:59Z")
My logstash runs inside a docker container and I mount the pattern via docker-compose. Everytime I edited the conf or pattern I removed the container and created a new one. Than I checked the files inside the container if they are correct. Can you tell me what permissions the pattern file needs? Maybe the user inside the container is not allowed the read the file? Maybe it helps when I create a new conf file and a new pattern file.. ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 22, 2021, 12:07pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/16 "2021-07-22T12:07:38Z")
So I installed a new Logstash instance without Elasticsearch and Kibana in a different vm and everything works fine with the sample log from above and follwoing configuration: Config: ```auto input { stdin { } } filter { grok { patterns_dir => ["/usr/share/logstash/patterns"] match => { "message" => "%{SYSLOGTIMESTAMP:tac_timestamp}\\t%{IPV4:IGP}\\t%{PPPOEUSER:pppoeuser}\\t%{INTERFACE:Interface}\\t%{GREEDYDATA:tac_message}" } } } output { stdout { codec => rubydebug } } ``` Custom Pattern: ```auto PPPOE [.a-zA-Z0-9_-]+ STATIC [.a-zA-Z0-9_-]+ PPPOEUSER %{PPPOE}@%{STATIC} INTERFACE \d/\d/\d/[\d]+ ``` Logstash output ```auto { "Interface" => "0/0/0/0000", "tac_timestamp" => "Mar 10 00:59:50", "tac_message" => "unknown\\tstart\\ttask_id=67511\\ttimezone=met\\tservice=ppp", "@timestamp" => 2021-07-22T12:07:03.640Z, "IGP" => "1.1.1.1", "@version" => "1", "pppoeuser" => "test@static", "host" => "ubuntu", "message" => "\"Mar 10 00:59:50\\t1.1.1.1\\ttest@static\\t0/0/0/0000\\tunknown\\tstart\\ttask_id=67511\\ttimezone=met\\tservice=ppp" } ``` ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [July 22, 2021, 12:12pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/17 "2021-07-22T12:12:12Z")
I think, if it is a problem of permission, an error will be write in logsatsh log file. You have to give permission to user or group `logstash` to access the file. `chmod -R logstash /path/to/patterns` If you can't change access to files, grok have an option named pattern\_definitions (example [here](https://discuss.elastic.co/t/grok-multiple-pattern-definitions/254797/2)). ---
**Author:** ![baumi](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@baumi](https://discuss.elastic.co/u/baumi)\ **Post date:** [July 22, 2021, 1:14pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/18 "2021-07-22T13:14:39Z")
I found my problem.. i was just looking at message field from Logstash output and I was trying to parse the message. But the acutal log which is getting shipped by Filebeat has a different format.. ```auto Jul 22 14:44:33 1.1.1.1 test@static 0/0/0/0000 unknown stop task_id=93491 timezone=mest service=ppp ``` sorry.. my bad ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 19, 2021, 1:15pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-grok-debugger-looks-good/279269/19 "2021-08-19T13:15:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.