Ingest node message parsing issue

Elastic Stack Beats
1

Hi ,
I have some share point log that that i want to use _ingest in order to process them.
its works perfect on the grok debugger but im unable to post it without parsing error

this is the grok pattern
(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) \t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}\t%{UUID:CorrelationID}

post it to the server as valid json is a bit tricky ...
so i tried to url encode the data

 "patterns": ["(%3F%3Cparsedtime%3E%25%7BMONTHNUM%7D%2F%25%7BMONTHDAY%7D%2F%25%7BYEAR%7D%20%25%7BHOUR%7D%3A%25%7BMINUTE%7D%3A%25%7BSECOND%7D)%20%5Ct%25%7BDATA%3Aprocess%7D%20%5C(%25%7BDATA%3Aprocesscode%7D%5C)(%5Cs*)%5Ct%25%7BDATA%3Atid%7D(%5Cs*)%5Ct(%3F%3Carea%3E.*)(%5Cs*)%5Ct(%3F%3Ccategory%3E.*)(%5Cs*)%5Ct%25%7BWORD%3AeventID%7D(%5Cs*)%5Ct%25%7BWORD%3Alevel%7D(%5Cs*)%5Ct%25%7BDATA%3Aeventmessage%7D%5Ct%25%7BUUID%3ACorrelationID%7D"]

        },

Any suggestion how to post correctly this pattern ?

the error thrown by the server

  "caused_by": {
                "type": "json_parse_exception",
                "reason": "Illegal unquoted character ((CTRL-CHAR, code 9)): has to be escaped using backslash to be included in string value\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@17e1d17e; line: 12, column: 39]"
            }
2

Could you please share the full pipeline?

3 
{
    "description": "sharepoint pipeline",
    "processors": [
        {
       "grok": {
        "field": "message",
         "patterns": ["(%3F%3Cparsedtime%3E%25%7BMONTHNUM%7D%2F%25%7BMONTHDAY%7D%2F%25%7BYEAR%7D%20%25%7BHOUR%7D%3A%25%7BMINUTE%7D%3A%25%7BSECOND%7D)%20%5Ct%25%7BDATA%3Aprocess%7D%20%5C(%25%7BDATA%3Aprocesscode%7D%5C)(%5Cs*)%5Ct%25%7BDATA%3Atid%7D(%5Cs*)%5Ct(%3F%3Carea%3E.*)(%5Cs*)%5Ct(%3F%3Ccategory%3E.*)(%5Cs*)%5Ct%25%7BWORD%3AeventID%7D(%5Cs*)%5Ct%25%7BWORD%3Alevel%7D(%5Cs*)%5Ct%25%7BDATA%3Aeventmessage%7D%5Ct%25%7BUUID%3ACorrelationID%7D"]

        }
      
        }
       
    ]
}
4

And also an example log. Sorry for not asking for it previously.

5 
Timestamp              	Process                                 	TID   	Area                          	Category                      	EventID	Level     	Message 	Correlation
09/03/2018 09:14:12.18 	w3wp.exe (0x31B4)                       	0x3F9C	SharePoint Foundation         	Runtime                       	afu6b	High    	[Forced due to logging gap, cached @ 09/03/2018 09:14:12.10, Original Level: VerboseEx] No SPAggregateResourceTally associated with thread.
6

The escaping was not correct in your pipeline. It's tricky to port patterns developed with Grok debugger to Filebeat. As it's read from a file using Golang slashes need extra escaping.

So your pattern should look like this:
(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) \\t%{DATA:process} \\(%{DATA:processcode}\\)(\\s*)\\t%{DATA:tid}(\\s*)\\t(?<area>.*)(\\s*)\\t(?<category>.*)(\\s*)\\t%{WORD:eventID}(\\s*)\\t%{WORD:level}(\\s*)\\t%{DATA:eventmessage}\\t%{UUID:CorrelationID}

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.