How to grok those multiline log syntax?

Elastic Stack Logstash
1

Hi,
I'm new to Elastic and after creating my first pipeline, I'm now trying to process a simple, local multiline Firewall log, it looks like this:

Time: 01/28/2022 01:27:22
Event: Traffic
IP-Address: 20.199.120.85
Description:
Path:
Message: Blocked Incoming TCP - Source 20.199.120.85 : https (443) Destination
192.168.0.158 : (49694)
Rule: Block all traffic

I have configured my filebeat.yml for multiline events but was not able to test it so far, because I'm failing completely when it comes to grokking the fields here with the Grok debugger.
Afaik, after getting treated as a multiline, all lines are packed into one single line.
The fields "Description: " and "Path: " are sometimes empty, but sometimes they contain something.
I started with something like this:
%{DATESTAMP:Time}%{GREEDYDATA}%{WORD:Event}%{GREEDYDATA}%{IPV4:IP}%{GREEDYDATA}%{WORD:Description}%{GREEDYDATA}%{WORD:Path}... and then gave up.

After hours of back and forth, I'm completely frustrated and I have now clue how to handle this. I guess I might need regex, but how? It sounds easy: Create a description field and ingest the following data. If fields like "Description: " and "Path: " are empty, leave them blank and continue.

I hope my explanation was somehow clear, I guess for you all this might be one of the easiest tasks ever, but everything I tried so far, failed.

I would be thankful if someone could help me out with this.

Thank you!

2

This should work

filter {
    # if you get by filebeat, grok should start with (?m)
    # grok {  match => { "message" => "(?m)Time....
	grok { 
		match => { "message" => "Time:\s*%{DATESTAMP:timestamp}\s*Event:\s*%{WORD:event}\s*IP-Address:\s*%{IP:ip}\s*Description:\s*%{DATA:description}\s*Path:\s*%{DATA:path}\s*Message:\s*%{DATA:msg}\s*Rule:\s*%{GREEDYDATA:rule}" }
	}
	
	date {
		match => ["timestamp", "MM/dd/yyyy HH:mm:ss"  ]
		remove_field => ["log", "host", "message","timestamp"] #, "tags"
	}
	
	if [ip]{
	  geoip {
		source => "[ip]"
		ecs_compatibility => "disabled"
	  }
	}

}
output {
 stdout {codec => rubydebug}
}
3

Another approach would be

    grok {
        break_on_match => false
        match => {
            message => [
                "^Time: (?<time>[^\n]+)$",
                "^Event: (?<event>[^\n]+)$",
                "^IP-Address: (?<ipAddress>[^\n]+)$",
                "^Description: (?<description>[^\n]+)$",
                "^Message: (?<msg>[^\n]+)$",
                "^Rule: (?<rule>[^\n]+)$"
            ]
        }
    }

which produces

       "msg" => "Blocked Incoming TCP - Source 20.199.120.85 : https (443) Destination",
 "ipAddress" => "20.199.120.85",
      "rule" => "Block all traffic",
      "time" => "01/28/2022 01:27:22"
1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.