I try to parse some simple log and get _grokparsefailure which I can't figure out.
Here's the log record example:
[DEBUG][444][Main]
It contains loglevel, pid and thread name. I configured rather simple pattern for this:
THREAD [a-zA-Z0-9_.-]+
MYPATTERN \[%{LOGLEVEL:loglevel}\]\[%{INT:pid}\]\[%{THREAD:thread}\]
And get an error:
{
"message" => "[DEBUG][444][Main]",
"@version" => "1",
"@timestamp" => "2016-08-05T15:54:43.769Z",
"path" => "/var/log/upstart/test.log",
"host" => "host-1",
"type" => "mylog",
"tags" => [
[0] "_grokparsefailure"
]
}
However, if I bring square brackets out into THREAD definition the pattern works fine:
THREAD \[[a-zA-Z0-9_.-]+)\]
MYPATTERN [%{LOGLEVEL:loglevel}][%{INT:pid}]%{THREAD:thread}
{
"message" => "[DEBUG][444][Main]",
"@version" => "1",
"@timestamp" => "2016-08-05T15:59:44.128Z",
"path" => "/var/log/upstart/test.log",
"host" => "host-1",
"type" => "mylog",
"loglevel" => "DEBUG",
"pid" => "444",
"thread" => "[Main]"
}
The thing is I don't need brackets in parsed field and want to get rid of them like in loglevel and pid fields.
The Grok Debugger also parses my custom pattern fine:
{
"MYLOG": [
[
"[INFO][20032][Main]"
]
],
"loglevel": [
[
"INFO"
]
],
"process": [
[
"20032"
]
],
"thread": [
[
"Main"
]
]
}
I wondered why loglevel and pid parsed properly. My though is that in pid and loglevel I used default grok patterns INT and LOGLEVEL, respectively. Then I tried to replace my THREAD regex by something default but similar, e.g. USERNAME, and it had been parsed! Then I tried to use the exact USERNAME regex in my THREAD pattern, and it failed.
My question is what should I do to parse a word in brackets not including brackets itselves?
I know I can replace brackets later by some other filter, but I think that this functionality fits the grok plugin.
My logstash version is 2.3.4 and grok version is 2.0.5.