Logstash Pipeline issue (I'm an idiot or something)

Elastic Stack Logstash
1

Hey gang. I am working with our dev team and they are dumping custom logging into the windows events
Using winlogbeat to ship the log entries into logstash i am trying to set up a central pipeline to handle the different api logs.
I am having trouble splitting up the data and it's all ending up in param1.

The windows log data is coming in like this:

Identifier: c27948fe-af6e-42e2-9db6-338f1d48f9e0
IdentifierType: Test LogEvent
ApplicationName: LoggingNuget-Tester
Severity: Debug
LogType: 
Name: Error talking to database
Description: Something went wrong in the operation.
FullStackTrace: Unable to find the specified file.
AuditCreatedDate: 12/17/2020 4:55:56 PM

Using the beats input and regular winlogs flow right through no problem to elasticsearch

My filter looks like this:

     if [winlog][channel] == "LoggingAPI" {
		if [message] =~ "Identifier: " 			{ grok { match => { "message" => "Identifier: %{DATA:log.identifier}" } } }
		if [message] =~ "IdentifierType: " 		{ grok { match => { "message" => "IdentifierType: %{DATA:log.identifierType}" } } }
		if [message] =~ "ApplicationName: "		{ grok { match => { "message" => "ApplicationName: %{DATA:log.applicationName}" } } }
		if [message] =~ "Severity: " 			{ grok { match => { "message" => "Severity: %{DATA:log.severity}" } } }
		if [message] =~ "LogType: " 			{ grok { match => { "message" => "LogType: %{DATA:log.logType}" } } }
		if [message] =~ "Name: " 				{ grok { match => { "message" => "Name: %{DATA:log.name}" } } }
		if [message] =~ "Description: " 		{ grok { match => { "message" => "Description: %{DATA:log.description}" } } }
		if [message] =~ "FullStackTrace: " 		{ grok { match => { "message" => "FullStackTrace: %{GREEDYDATA:log.fullStackTrace}" } } }
        mutate{ add_tag => [ "logging-api-via-logstash" ]}
    }

The only line that makes it into it's own field is the last one "FullStackTrace" (of course that brings along the darn AuditCreatedDate

If I try to use GREEDYDATA on the other lines it pulls the next line or more of data.
I don't think I understand on how to get the value but stop at the new line.
These logging events should never take up more than one line .. except of course the fullstacktrace.

What am I doing wrong here?
Thanks

2

DATA can match a lot or a little, or even nothing at all. Try enabling the grok option keep_empty_captures => true. If you then find you get the fields that you expect but they are empty strings then try changing the patterns to things like "message" => "Identifier: %{DATA:log.identifier}$" to force DATA to consume the rest of the line. Then decide if you want to retain keep_empty_captures (i.e., do you want a LogType field for your example data).

If that is your issue then it definitely does not indicate you are an idiot. It took me years to understand the greedy/not-greedy distinction for a Ruby regexp. As far as I know, the other regexp languages I have used for decades (ed, sh, csh, awk, and others) are basically greedy by default. GREEDYDATA will match as much as possible. DATA will match as little as possible, and if DATA is at the end of the pattern it can return a positive match for zero characters. You need a way to force DATA to consume the rest of the line. The $ I used above does that. Another way of consuming the line is a custom pattern that matches zero or more not-newline "message" => "Identifier: (?<log.identifier>[^\n]*)".

Note that log.identifier will be a field with a period in its name. If your intention was to create an object like

"log": { "identifier": "...", "name": "...", ... }

then you would need to use [log][identifier] etc.

1 Like
3

Thanks Badger.
I'll try this out now

4

Ok great! I was able to get the results you detailed. and exactly as you detailed.. the first result were the fields albeit empty.. once I added the $'s they were populated with the expected data.

I'll still need to do a bit of tweaking but I thank you very VERY much for steering me the right way..
Unfortunately I have to wear many hats and my grok hat is waaaay too small and I don't wear it all that often.

I'll be back for more soon I'm sure.

5

Looking good, Thanks again!

    log.application_name    LoggingNuget-Tester
    log.audit_created_date    12/17/2020 8:03:01 PM
    log.description    Help, I'm trapped in this testing program!
    log.full_stack_trace    Unable to find the specified file.
    log.identifier    8e8ba83f-3664-4e5b-ad74-433a0a81fffd
    log.identifier_type    Test LogEvent
    log.level    information
    log.name    LoggingNuget-Tester
    log.severity    Trace
    
    message
    Identifier: 8e8ba83f-3664-4e5b-ad74-433a0a81fffd
    IdentifierType: Test LogEvent
    ApplicationName: LoggingNuget-Tester
    Severity: Trace
    LogType: 
    Name: Standard operation
    Description: Help, I'm trapped in this testing program!
    FullStackTrace: Unable to find the specified file.
    AuditCreatedDate: 12/17/2020 8:03:01 PM
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.