_grokparsefailure but it works fine in dubugger

miovinelli · March 22, 2018, 4:21pm

Hi all,
there is a strange situation cause my pattern works fine in grokdebug.herokuapp.com and also in a debugger offered by X-Pack, but at runtime ingestion events are tagged with "_grokparsefailure" and I don't understand why. Please can someone help me? Thanks in advance

pipeline conf:
input {
beats {
port => "5043"
}
}
filter {
if "WLS-log" in [tags] {
grok {
patterns_dir => "./patterns"
match => [ "message", "<%{TIMESTAMP_ISO8601:timestamp_log}>\s-\s%{DATA:element}\s-\s%{DATA:id}\s-\s%{DATA:log_level}\s\s%{MSG:msglog}?" ]

}
mutate {
  	add_field => { "Application" => "WLS-INTRA" }
  	add_field => { "log_type" => "service-log" }
}

}
}

Pattern:
MSG (.|\r|\n|.)

Follow 3 lines of our log file:

<2018-03-21 14:33:16.717> - SEU - [rJFIp41ZPB4Oc0T0dxkGAjj0Eza2Z4SVW46HEMkaKxuEgpDc6fwr!-172047753!1521637363033] - INFO it.gse.seu.web.task.TasksBean - creating task filters for intranet user: a852197seu
<2018-03-21 14:33:33.056> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in
LdapUser {
userId=a852514,
name=Mariachiara,
surname=Cusano,
email=mariachiara.cusano@gse.it,
officeName=Sistemi di Produzione e Consumo,
roles=[TL_INT, ASSIGNER]
}
<2018-03-21 14:33:33.171> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO it.gse.seu.web.task.TasksBean - creating task filters for intranet user: a852514

Badger · March 22, 2018, 4:39pm

I cannot speak to why grok is not working, but I would do that using dissect.

dissect { mapping => { "message" => "<%{ts} %{+ts}> - %{element} - %{id} - %{log_level} %{msglog} " } }

vitich · March 22, 2018, 4:53pm

Had the same problem...
Try patterns_dir => "./patterns/YOUR_FILE_WITH_PATTERNS" instead.
Worked for me.

miovinelli · March 23, 2018, 8:32am

Thanks but result is the same..._grokparsefailure

Christian_Dahlqvist · March 23, 2018, 8:42am

Are you sure you always have exactly 2 spaces ahead of %{MSG:msglog} ?

miovinelli · March 23, 2018, 8:57am

Thanks but result is the same. _grokparsefailure

Christian_Dahlqvist · March 23, 2018, 9:01am

I would recommend configuring a stdout output plugin with a rubydebug codec to troubleshoot this. It would probably help if you could show us the exact result for your 3 example events. make sure you format the data correctly using the UI tools.

vitich · March 23, 2018, 9:22am

Did you check the pattern on these sites?

http://grokconstructor.appspot.com/do/match#result
https://grokdebug.herokuapp.com/

miovinelli · March 23, 2018, 11:37am

Stdout doesn't help us with more info.....

{
"log_level" => "INFO",
"id" => "[y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029]",
"Application" => "WLS-INTRA",
"timestamp_log" => "2018-03-21 14:33:33.056",
"message" => "<2018-03-21 14:33:33.056> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in",
"@version" => "1",
"offset" => 400,
"host" => "MacBookPro.local",
"msglog" => " i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in",
"log_type" => "service-log",
"@timestamp" => 2018-03-23T11:01:24.388Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied"
],
"prospector" => {
"type" => "log"
},
"element" => "SEU",
"beat" => {
"hostname" => "MacBookPro.local",
"version" => "6.2.2",
"name" => "MacBookPro.local"
},
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log"
}
{
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"message" => " LdapUser {",
"@version" => "1",
"offset" => 412,
"host" => "MacBookPro.local",
"log_type" => "service-log",
"@timestamp" => 2018-03-23T11:01:24.388Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"beat" => {
"version" => "6.2.2",
"hostname" => "MacBookPro.local",
"name" => "MacBookPro.local"
},
"Application" => "WLS-INTRA"
}

miovinelli · March 23, 2018, 11:42am

Pattern works fine with https://grokdebug.herokuapp.com/ but not with http://grokconstructor.appspot.com/do/match#result. Below there is a part of the output

MATCHED
element SEU
log_level INFO
msglog ·i.g.s.w.a.IntranetAuthenticationFilter·-·user:·Mariachiara·is·logged·in
id [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029]
timestamp_log 2018-03-21·14:33:33.056
LdapUser {
NOT MATCHED. The longest regex prefix matching the beginning of this line is as follows:

prefix
after match: LdapUser {
userId=a852514,

Christian_Dahlqvist · March 23, 2018, 11:48am

This event seem to parse fine.

When one looks at this event it however looks like you are not handling the multiline event correctly, which is why grok fails.

miovinelli · March 23, 2018, 2:11pm

Yes is true, it's multiline case, but should be allow cause my extra pattern MSG is defined like:

MSG (.|\r|\n|\t|\s|\f|\v|\x20|\N)*

Christian_Dahlqvist · March 23, 2018, 2:13pm

If you do not have multiline processing configured correctly in Filebeat, each line will come in as a separate event (which seems to be the case based on the output you provided), in which case it grok will never see the full event.

miovinelli · March 23, 2018, 2:32pm

Right clue! I add these new lines to filebeat.yml

multiline.pattern: '^<'
multiline.negate: true
multiline.match: after

but still the same....

Christian_Dahlqvist · March 23, 2018, 2:38pm

That looks correct. Are the lines now merged into full events? Can you show us the output?

miovinelli · March 23, 2018, 2:44pm

{
"host" => "MacBookPro.local",
"offset" => 206,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => " LdapUser {",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 297,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\temail=mariachiara.cusano@gse.it, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 244,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tname=Mariachiara, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 369,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\troles=[TL_INT, ASSIGNER]",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"log_level" => "INFO",
"offset" => 194,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"id" => "[y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029]",
"message" => "<2018-03-21 14:33:33.056> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log",
"element" => "SEU",
"timestamp_log" => "2018-03-21 14:33:33.056",
"msglog" => " i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in"
}
{
"host" => "MacBookPro.local",
"offset" => 262,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tsurname=Cusano, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 224,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tuserId=a852514, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"prospector" => {
"type" => "log"
},
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 343,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tofficeName=Sistemi di Produzione e Consumo, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}

Christian_Dahlqvist · March 23, 2018, 2:46pm

It does not look like your multiline config is working.

miovinelli · March 23, 2018, 2:47pm

filebeat.yml is:

filebeat.prospectors:

####### OHS INTRA LOG #######
- type: log
  tags: ["OHS-INTRA-access-log"]
  paths:
    - /Users/kp/Progetti/GSE/input da Cliente/*access*log
    
    
####### WLS LOG #######
- type: log
  tags: ["WLS-log"]
  paths:
    - /Users/kp/Progetti/GSE/input da Cliente/seu*.log
    
####### OAM LOG #######
- type: log
  tags: ["OAM-log"]
  paths:
    - /Users/kp/Progetti/GSE/input da Cliente/*diagnostic.log

multiline.pattern: '^\<'
multiline.negate: true
multiline.match: after

output.logstash:
  hosts: ["localhost:5043"]

Christian_Dahlqvist · March 23, 2018, 2:49pm

Please format that correctly as yml files are sensitive to indentation. I believe the multiline config should be under each prospector it applies to, which does not necessarily seem to be the case here.

miovinelli · March 23, 2018, 2:53pm

Works fine with this configuration:

filebeat.prospectors:

####### OHS INTRA LOG #######

type: log
tags: ["OHS-INTRA-access-log"]
paths:
- /Users/kp/Progetti/GSE/input da Cliente/accesslog

####### WLS LOG #######

type: log
tags: ["WLS-log"]
paths:
- /Users/kp/Progetti/GSE/input da Cliente/seu*.log
multiline.pattern: '^<'
multiline.negate: true
multiline.match: after

####### OAM LOG #######

type: log
tags: ["OAM-log"]
paths:
- /Users/kp/Progetti/GSE/input da Cliente/*diagnostic.log

output.logstash:
hosts: ["localhost:5043"]

Thank you so much for your help. Really appreciate! Thanks again

Topic		Replies	Views
_Grokparsefailure even though pattern works in Grok Debugger Logstash	3	787	October 29, 2018
"_grokparsefailure" even though the grok pattern matches Logstash	9	4363	September 11, 2017
_grokparsefailure problem Logstash	3	357	September 4, 2019
"_grokparsefailure" for pattern that works on grok debugger Logstash	3	688	July 16, 2020
Logstash '_grokparsefailure' issue Logstash	14	36771	July 6, 2017

_grokparsefailure but it works fine in dubugger

Related topics