Logstash - Grok parse failure

[Aby (Abisheik)]

Hi - i am trying to parse the below log file:

[2018-05-29 08:04:23,687] DEBUG - http-outgoing-2845 << HTTP/1.1 500 Internal Server Error {org.apache.synapse.transport.http.headers}
[2018-05-29 08:04:18,037] DEBUG - http-outgoing-2842 << HTTP/1.1 200 OK {org.apache.synapse.transport.http.headers}

below is my conf

input {
file {
path => "/home/abhi/ipclick.log"
start_position=> "beginning"
}
}

filter {
grok {
break_on_match => false
match =>
{'message' => ['%{SYSLOG5424SD:date} DEBUG - http-outgoing%{BASE16FLOAT:outputid} << HTTP%{URIPATHPARAM:URL} 500 %{GREEDYDATA:STATUS} {%{JAVACLASS:MSG}}','%{SYSLOG5424SD:date} DEBUG - >> %{QS:STATUS} {%{JAVACLASS:MSG}}','%{SYSLOG5424SD:date} DEBUG - http-outgoing%{BASE16FLOAT:outputid} >> %{GREEDYDATA: request} %{URIPATHPARAM:URL} {%{JAVACLASS:MSG}}'
,'%{SYSLOG5424SD:date} DEBUG - http-outgoing%{BASE16FLOAT:outputid} << HTTP%{URIPATHPARAM:URL} 200 %{GREEDYDATA:STATUS} {%{JAVACLASS:MSG}}']}
}

#if "_grokparsefailure" in [tags] {

drop{}

#}
}
output {
elasticsearch {
hosts => "XXXXX:9200"
index => "log"
}
stdout {
codec => rubydebug
}
}

I get the below output: [even though my message pattern is same for output 1 & 2, my second output parses the value but with a _grokparsefailure issue.

grok filter pattern that i have used in both the cases are same. Any help will be appreciated..

{
"URL" => "/1.1",
"path" => "/home/abhi/ipclick.log",
"MSG" => "org.apache.synapse.transport.http.headers",
"host" => "SCPUBU",
"date" => "[2018-05-29 08:04:18,037]",
"outputid" => "-2842",
"message" => "[2018-05-29 08:04:18,037] DEBUG - http-outgoing-2842 << HTTP/1.1 200 OK {org.apache.synapse.transport.http.headers}",
"STATUS" => "OK",
"@timestamp" => 2018-07-03T09:59:22.087Z,
"@version" => "1"
}
{
"URL" => "/1.1",
"outputid" => "-2845",
"date" => "[2018-06-29 08:04:23,687]",
"message" => "[2018-06-29 08:04:23,687] DEBUG - http-outgoing-2845 << HTTP/1.1 500 OK {org.apache.synapse.tranr}",
"@timestamp" => 2018-07-03T09:59:22.086Z,
"@version" => "1",
"MSG" => "org.apache.synapse.tranr",
"path" => "/home/abhi/ipclick.log",
"host" => "SCPUBU",
"STATUS" => "OK",
"tags" => [
[0] "_grokparsefailure"
]
}

[Aby (Abisheik)]

Just to add:

In our case, there are 2 patterns defined in the array, only the second pattern works. 1st pattern throws a grok parse failure though it parses correctly without issue and this is the case if it's vice versa.

1st pattern always comes with the _grokparsefailure error

'message' => ['%{SYSLOG5424SD:date} DEBUG - http-outgoing-%{BASE16FLOAT:outputid} << HTTP%{URIPATHPARAM:URL} %{GREEDYDATA:STATUS}{%{JAVACLASS:MSG}}','%{SYSLOG5424SD:date} DEBUG - >> %{QS:STATUS} {%{JAVACLASS:MSG}}']

Output:
{
"@version" => "1",
"host" => "SCPUBU",
"STATUS" => "200 OK ",
"URL" => "/1.1",
"MSG" => "org.apache.synapse.transport.http.headers",
"tags" => [
[0] "_grokparsefailure"
],
"@timestamp" => 2018-07-03T16:27:32.822Z,
"date" => "[2018-05-29 08:04:18,037]",
"path" => "/home/abhi/test.log",
"message" => "[2018-05-29 08:04:18,037] DEBUG - http-outgoing-2842 << HTTP/1.1 200 OK {org.apache.synapse.transport.http.headers}",
"outputid" => "2842"
}

{
"@version" => "1",
"date" => "[2018-05-29 08:04:11,797]",
"host" => "SCPUBU",
"STATUS" => ""OPTIONS /intrCaco2.1.1/2.1.1/api/service/predict HTTP/1.1[\r][\n]"",
"path" => "/home/abhi/test.log",
"message" => "[2018-05-29 08:04:11,797] DEBUG - >> "OPTIONS /intrCaco2.1.1/2.1.1/api/service/predict HTTP/1.1[\r][\n]" {org.apache.synapse.transport.http.wire}",
"MSG" => "org.apache.synapse.transport.http.wire",
"@timestamp" => 2018-07-03T16:27:32.822Z
}