Grok filter Mappings ignored by logstash

Elastic Stack Logstash
1

we have created below grok filter for mapping the log fields, but still logstash is using default mapping when pipeline initiated.
//
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} +%{LOGLEVEL:severity} +%{JAVACLASS:class}: +allowed=%{WORD:allowed} +ugi=%{USERNAME:ugi} +(auth:%{WORD:auth}) +via +%{USERNAME:ugi2} +(auth:%{WORD:auth2}) +ip=/%{IP:ip} +cmd=%{WORD:cmd} +src=%{PATH:src} +dst=%{USERNAME:dst} +perm=%{USERNAME:perm} +proto=%{USERNAME:proto} +callerContext=%{GREEDYDATA:callerContext}" }
add_field => {
"type" => "pattern1"
}
}
grok {
match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp} +%{LOGLEVEL:severity} +%{JAVACLASS:class}: +allowed=%{WORD:allowed} +ugi=%{USERNAME:ugi} +(auth:%{WORD:auth}) +ip=/%{IP:ip} +cmd=%{WORD:cmd} +src=%{PATH:src} +dst=%{USERNAME:dst} +perm=%{USERNAME:perm} +proto=%{USERNAME:proto}"}
add_field => {
"type" => "pattern2"
}
}
}
//

mapping in results are generated like below (but it should follow the mapping provided in filter ?) :=>
"@version" => "1",
"tags" => [
[0] "_grokparsefailure"
],
"path" => "/var/log/hadoop/hdfs/hdfs-audit.log.2018-07-26",
"message" => "2018-07-26 23:59:57,497 INFO FSNamesystem.audit: allowed=true\tugi=yarn (auth:SIMPLE)\tip=/10.155.140.194\tcmd=listStatus\tsrc=/ats/active\tdst=null\tperm=null\tproto=rpc\tcallerContext=yarn_ats_server_v1_5",
"@timestamp" => 2018-07-31T07:01:56.905Z,
"host" => "CTSC00557182401"

2

Your grok patterns appear to match spaces using ' +', but your event contains tabs '\t'.

3

Thanks for the reply badger , that's added because "message" field was interpreting multi space with "\t" to mitigate I used multiple grok filters (one more with "+") but still it's prompting default.

like :
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} +%{LOGLEVEL:severity} +%{JAVACLASS:class}: +allowed=%{WORD:allowed} +ugi=%{USERNAME:ugi} +(auth:%{WORD:auth}) +via +%{USERNAME:ugi2} +(auth:%{WORD:auth2}) +ip=/%{IP:ip} +cmd=%{WORD:cmd} +src=%{PATH:src} +dst=%{USERNAME:dst} +perm=%{USERNAME:perm} +proto=%{USERNAME:proto} +callerContext=%{GREEDYDATA:callerContext}"}
}
grok {
# tab with callerContext with via
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} +%{LOGLEVEL:severity} +%{JAVACLASS:class}: +allowed=%{WORD:allowed}\tugi=%{USERNAME:ugi} +(auth:%{WORD:auth}) +via +%{USERNAME:ugi2} +(auth:%{WORD:auth2})\tip=/%{IP:ip}\tcmd=%{WORD:cmd}\tsrc=%{PATH:src}\tdst=%{USERNAME:dst}\tperm=%{USERNAME:perm}\tproto=%{USERNAME:proto}\tcallerContext=%{GREEDYDATA:callerContext}"}
}

}

4

Please show both the message and the grok pattern indented by four spaces, so that escapes are shown.

Like this:

"message" => "2018-07-26 23:59:57,497 INFO FSNamesystem.audit: allowed=true\tugi=yarn (auth:SIMPLE)\tip=/10.155.140.194\tcmd=listStatus\tsrc=/ats/active\tdst=null\tperm=null\tproto=rpc\tcallerContext=yarn_ats_server_v1_5",

Not like this

"message" => "2018-07-26 23:59:57,497 INFO FSNamesystem.audit: allowed=true\tugi=yarn (auth:SIMPLE)\tip=/10.155.140.194\tcmd=listStatus\tsrc=/ats/active\tdst=null\tperm=null\tproto=rpc\tcallerContext=yarn_ats_server_v1_5",

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.