Grokprasefailure Issue with hyphen in the log

Elastic Stack Logstash
#1

I am getting a grokparsefailure message for the below stated conf. I'm not getting the SeverityLevel. The problem for with the - ( hyphen ) in the log. NOTSPACE isn't working for me. I would appreciate any help.

Input Log:

{"message":"<187><187> Oct 04 21:32:18 apic1 %LOG_LOCAL7-3" }

filter {
grok {
match => { "message" => "<%{NUMBER:STARTCODE01}><%{NUMBER:STARTCODE2}>%{SPACE}%{SYSLOGTIMESTAMP:LogTimeStamp}%{SPACE}%{WORD:MessageSource}%{SPACE}%LOG_LOCAL%{NUMBER:FacilityLevel}%{NOTSPACE}%{NUMBER:SeverityLevel}" }
}
}

#2

You are applying the json filter first, I hope, and not trying to parse JSON with a grok filter?

This one wasn't obvious, but the problem turned out to be the definition of the NUMBER pattern:

BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})

Note how BASE10NUM begins with a negative lookbehind; it only matches if it's not preceded by a digit, decimal point, plus, or minus.

Use INT instead of NUMBER. The severity will never be a float anyway so there's no reason to use NUMBER in the first place.

#3

Thanks a lot Magnus. That helped. I'm using a text file with logs in it as the input for my logstash config. Just trying to use grok to match the format of the log. More like a practice . I need your suggestion to work out this,

[uni/ten-[uni/tn-DevVFtd59]-scriptHandlerState/LDevOpInf-Dev]

I would like to store the entire information between the brackets in one field. Something like,

"Field_name" = "uni/ten-[uni/tn-DevVFtd59]-scriptHandlerState/LDevOpInf-Dev"

I'm using match of grok to do this. I tried using GREEDYDATA to get this done but, it didn't work. It replaced values of other fields that I've stored ahead of this one.

So, I've tried to match everything inside the brackets and created multiple fields to store information. It would be so easy for me if I could store the entire thing inside the brackets in to one field. Could you please suggest something here.

Example :

{"message":"<190><190> Oct 04 21:26:11 apic1 %LOG_LOCAL7-6-SYSTEM_MSG [E4207683][transition][info][uni/ten-[uni/tn-DevVFtd59]-scriptHandlerState/LDevOpInf-Dev] LDevOperInfo Dev modified"}

My Filter:

grok {
match => { "message" => "<%{NUMBER:STARTCODE01}><%{NUMBER:STARTCODE2}> %{SYSLOGTIMESTAMP:LogTimeStamp} %{WORD:MessageSource} %LOG_LOCAL%{INT:FacilityLevel}%{NOTSPACE}%{INT:SeverityLevel}%{NOTSPACE}SYSTEM%{NOTSPACE}MSG %{NOTSPACE}%{BASE16NUM:ErrorCode}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{WORD:LifeCycleState}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{WORD:ITULevel}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{WORD:RootName}%{NOTSPACE}%{WORD:SubFolderName}%{NOTSPACE}%{NOTSPACE}%{WORD:RootName}%{NOTSPACE}%{WORD:SubFolderName}%{NOTSPACE}%{WORD:TenantName}%{NOTSPACE}%{NOTSPACE}%{WORD:StateName}%{NOTSPACE}%{WORD:LDevOpInf}%{NOTSPACE}%{WORD:LDevName}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Message}"}
}

Thanks.

#4

You can use [^\]] to match any character except closing square bracket, but the nested square bracket pairs make this more difficult. Will there ever be spaces inside the string? You should be able to use a zero-length positive lookahead to check that the closing square bracket must be following be a space. Or just do this:

\[info\]\[%{NOTSPACE:whatever}\] LDevOperInfo

(Yeah, I realize that "info" and "LDevOperInfo" aren't fixed strings but I thought it would make the example clearer.)

Any particular reason you're using %{SPACE} instead of a literal space or \s? Your expression is very tedious to read.

#5

Sorry, no particular reason for %{SPACE}. I've cleaned my config. Lucky that there's no spacing at all inside those brackets in any of the logs. I was able to get the whole thing into a field. Thanks a lot man. Working on parsing the field now. Here's my current config.

match =>  { "message" => "<%{NUMBER:STARTCODE01}><%{NUMBER:STARTCODE2}> %{SYSLOGTIMESTAMP:LogTimeStamp} %{WORD:MessageSource} %LOG_LOCAL%{INT:FacilityLevel}\-%{INT:SeverityLevel}\-SYSTEM\_MSG \[%{BASE16NUM:ErrorCode}\]\[%{WORD:LifeCycleState}\]\[%{WORD:ITULevel}\]\[%{NOTSPACE:AffectedDN}\] %{GREEDYDATA:Rest}"}

Let me know if I need to improve on something. Is it okay If I use the %{GREEDYDATA:Rest} to get the message?

#6

Using GREEDYDATA at the end of the expression is fine.

#7

Thanks Magnus. I really appreciate your support.

#8