_grokparsefailure but it works fine in dubugger

Hi all,
there is a strange situation cause my pattern works fine in grokdebug.herokuapp.com and also in a debugger offered by X-Pack, but at runtime ingestion events are tagged with "_grokparsefailure" and I don't understand why. Please can someone help me? Thanks in advance

pipeline conf:
input {
beats {
port => "5043"
}
}
filter {
if "WLS-log" in [tags] {
grok {
patterns_dir => "./patterns"
match => [ "message", "<%{TIMESTAMP_ISO8601:timestamp_log}>\s-\s%{DATA:element}\s-\s%{DATA:id}\s-\s%{DATA:log_level}\s\s%{MSG:msglog}?" ]

}
mutate {
  	add_field => { "Application" => "WLS-INTRA" }
  	add_field => { "log_type" => "service-log" }
}

}
}

Pattern:
MSG (.|\r|\n|.)

Follow 3 lines of our log file:

<2018-03-21 14:33:16.717> - SEU - [rJFIp41ZPB4Oc0T0dxkGAjj0Eza2Z4SVW46HEMkaKxuEgpDc6fwr!-172047753!1521637363033] - INFO it.gse.seu.web.task.TasksBean - creating task filters for intranet user: a852197seu
<2018-03-21 14:33:33.056> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in
LdapUser {
userId=a852514,
name=Mariachiara,
surname=Cusano,
email=mariachiara.cusano@gse.it,
officeName=Sistemi di Produzione e Consumo,
roles=[TL_INT, ASSIGNER]
}
<2018-03-21 14:33:33.171> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO it.gse.seu.web.task.TasksBean - creating task filters for intranet user: a852514

I cannot speak to why grok is not working, but I would do that using dissect.

dissect { mapping => { "message" => "<%{ts} %{+ts}> - %{element} - %{id} - %{log_level} %{msglog} " } }

Had the same problem...
Try patterns_dir => "./patterns/YOUR_FILE_WITH_PATTERNS" instead.
Worked for me.

Thanks but result is the same..._grokparsefailure

Are you sure you always have exactly 2 spaces ahead of %{MSG:msglog} ?

Thanks but result is the same. _grokparsefailure

I would recommend configuring a stdout output plugin with a rubydebug codec to troubleshoot this. It would probably help if you could show us the exact result for your 3 example events. make sure you format the data correctly using the UI tools.

Did you check the pattern on these sites?

http://grokconstructor.appspot.com/do/match#result
https://grokdebug.herokuapp.com/

Stdout doesn't help us with more info.....

{
"log_level" => "INFO",
"id" => "[y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029]",
"Application" => "WLS-INTRA",
"timestamp_log" => "2018-03-21 14:33:33.056",
"message" => "<2018-03-21 14:33:33.056> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in",
"@version" => "1",
"offset" => 400,
"host" => "MacBookPro.local",
"msglog" => " i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in",
"log_type" => "service-log",
"@timestamp" => 2018-03-23T11:01:24.388Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied"
],
"prospector" => {
"type" => "log"
},
"element" => "SEU",
"beat" => {
"hostname" => "MacBookPro.local",
"version" => "6.2.2",
"name" => "MacBookPro.local"
},
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log"
}
{
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"message" => " LdapUser {",
"@version" => "1",
"offset" => 412,
"host" => "MacBookPro.local",
"log_type" => "service-log",
"@timestamp" => 2018-03-23T11:01:24.388Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"beat" => {
"version" => "6.2.2",
"hostname" => "MacBookPro.local",
"name" => "MacBookPro.local"
},
"Application" => "WLS-INTRA"
}

Pattern works fine with https://grokdebug.herokuapp.com/ but not with http://grokconstructor.appspot.com/do/match#result. Below there is a part of the output

MATCHED
element SEU
log_level INFO
msglog ·i.g.s.w.a.IntranetAuthenticationFilter·-·user:·Mariachiara·is·logged·in
id [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029]
timestamp_log 2018-03-21·14:33:33.056
LdapUser {
NOT MATCHED. The longest regex prefix matching the beginning of this line is as follows:

prefix
after match: LdapUser {
userId=a852514,

This event seem to parse fine.

When one looks at this event it however looks like you are not handling the multiline event correctly, which is why grok fails.

Yes is true, it's multiline case, but should be allow cause my extra pattern MSG is defined like:

MSG (.|\r|\n|\t|\s|\f|\v|\x20|\N)*

If you do not have multiline processing configured correctly in Filebeat, each line will come in as a separate event (which seems to be the case based on the output you provided), in which case it grok will never see the full event.

Right clue! I add these new lines to filebeat.yml

multiline.pattern: '^<'
multiline.negate: true
multiline.match: after

but still the same....

That looks correct. Are the lines now merged into full events? Can you show us the output?

{
"host" => "MacBookPro.local",
"offset" => 206,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => " LdapUser {",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 297,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\temail=mariachiara.cusano@gse.it, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 244,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tname=Mariachiara, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 369,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\troles=[TL_INT, ASSIGNER]",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"log_level" => "INFO",
"offset" => 194,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"id" => "[y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029]",
"message" => "<2018-03-21 14:33:33.056> - SEU - [y_FIw8fl31L85jubyIyZLRASen-7ja_9GjM7C-th61F-YrJwxaoS!-172047753!1521639213029] - INFO i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log",
"element" => "SEU",
"timestamp_log" => "2018-03-21 14:33:33.056",
"msglog" => " i.g.s.w.a.IntranetAuthenticationFilter - user: Mariachiara is logged in"
}
{
"host" => "MacBookPro.local",
"offset" => 262,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tsurname=Cusano, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 224,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tuserId=a852514, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"prospector" => {
"type" => "log"
},
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}
{
"host" => "MacBookPro.local",
"offset" => 343,
"beat" => {
"name" => "MacBookPro.local",
"hostname" => "MacBookPro.local",
"version" => "6.2.2"
},
"message" => "\tofficeName=Sistemi di Produzione e Consumo, ",
"@timestamp" => 2018-03-23T14:43:27.991Z,
"tags" => [
[0] "WLS-log",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
],
"prospector" => {
"type" => "log"
},
"Application" => "WLS-INTRA",
"source" => "/Users/kp/Progetti/GSE/input da Cliente/seu.log",
"@version" => "1",
"log_type" => "service-log"
}

It does not look like your multiline config is working.

filebeat.yml is:

filebeat.prospectors:

####### OHS INTRA LOG #######
- type: log
  tags: ["OHS-INTRA-access-log"]
  paths:
    - /Users/kp/Progetti/GSE/input da Cliente/*access*log
    
    
####### WLS LOG #######
- type: log
  tags: ["WLS-log"]
  paths:
    - /Users/kp/Progetti/GSE/input da Cliente/seu*.log
    
####### OAM LOG #######
- type: log
  tags: ["OAM-log"]
  paths:
    - /Users/kp/Progetti/GSE/input da Cliente/*diagnostic.log

multiline.pattern: '^\<'
multiline.negate: true
multiline.match: after

output.logstash:
  hosts: ["localhost:5043"]

Please format that correctly as yml files are sensitive to indentation. I believe the multiline config should be under each prospector it applies to, which does not necessarily seem to be the case here.

Works fine with this configuration:

filebeat.prospectors:

####### OHS INTRA LOG #######

  • type: log
    tags: ["OHS-INTRA-access-log"]
    paths:
    • /Users/kp/Progetti/GSE/input da Cliente/accesslog

####### WLS LOG #######

  • type: log
    tags: ["WLS-log"]
    paths:

    • /Users/kp/Progetti/GSE/input da Cliente/seu*.log

    multiline.pattern: '^<'
    multiline.negate: true
    multiline.match: after

####### OAM LOG #######

  • type: log
    tags: ["OAM-log"]
    paths:
    • /Users/kp/Progetti/GSE/input da Cliente/*diagnostic.log

output.logstash:
hosts: ["localhost:5043"]

Thank you so much for your help. Really appreciate! Thanks again