Snort Logs with FileBeat

[johndowe]

Hi,

I have setup filebeat on a pi running Snort sending logs to a cloud ELK stack. I am trying to figure out how to arrange logs and doing the following process:

on the beats side i have this in the filebeat.yml:
paths:
- /var/log/snort/alert
tags: ["snort"]

Whilst in the filter on the logstash side I have the below:

if "snort" in [tags] {

# parse the message into individual fields
grok {
    match => { "message" => "(?<ts>.*\d{2}:\d{2}:\d{2})\s(?<host>.*?)\s.*?\s\[(?<generator_id>.*?)::(?<signature_id>.*?):.*?\]\s(?<signature>.*?)\s\[Classification:\s(?<classification>.*?)\]\s\[Priority:\s(?<priority>.*?)\].*?{(?<protocol>.*?)\}\s(?<source_ip>.*?):(?<source_port>.*?)\s-\>\s(?<destination_ip>.*?):(?<destination_port>.*)" }
}

# remove the original message if parsing was successful
if !("_grokparsefailure" in [tags]) {
    mutate {
        remove_field => [ "message" ]
    }
}

# parse the timestamp and save in a new datetime field
if [ts] {
    date {
        match => [ "ts", "MMM dd HH:mm:ss" ]
        target => "sys_timestamp"
    }

    # remove the original timestamp if date parsing was successful
    if !("_dateparsefailure" in [tags]) {
        mutate {
            remove_field => [ "ts" ]
        }
    }
}

}

A sample log file in snort looks as such:
04/22-17:47:29.436774 [] [1:29456:2] PROTOCOL-ICMP Unusual PING detected [] [Classification: Information Leak] [Priority: 2] {ICMP} 192.168.4.1 -> 192.168.4.9

Unfortunately the above is not working and noticed a tag in Kibana with snort logs as follows: _grokparsefailure.

Any thoughts on making this work?

[steffens (Steffen Siering)]

I moved it to the Logstash forums.

[Badger]

Firstly, your pattern first breaks at (?.*\d{2}:\d{2}:\d{2})\s since that does not match the microsecond part of the timestamp.

Secondly, application of the common options like remove_field, which is referred to as decoration, only occurs if the filter successfully executes. Thus

grok {
    match => { "message" => "(?<ts>.*\d{2}:\d{2}:\d{2})\..." }
}
# remove the original message if parsing was successful
if !("_grokparsefailure" in [tags]) {
    mutate { remove_field => [ "message" ] }
}

is exactly equivalent to

grok {
    match => { "message" => "(?<ts>.*\d{2}:\d{2}:\d{2})\..." }
    mutate { remove_field => [ "message" ] }
}

Thirdly, I would not use grok at all to parse that line, I would use dissect.

dissect { mapping => { "message" => "%{ts} [%{field1}] [%{field2}] %{signature} [%{field5}] [Classification: %{classification}] [Priority: %{priority}] %{restOfLine}" } }

would get you

    "restOfLine" => "{ICMP} 192.168.4.1 ->\n192.168.4.9",
        "field1" => "",
        "field5" => "",
    "@timestamp" => 2019-04-23T22:02:44.738Z,
     "signature" => "PROTOCOL-ICMP Unusual PING detected",
      "priority" => "2",
"classification" => "Information Leak",
       "message" => "04/22-17:47:29.436774 [] [1:29456:2] PROTOCOL-ICMP Unusual PING detected [] [Classification: Information Leak] [Priority: 2] {ICMP} 192.168.4.1 ->\n192.168.4.9",
        "field2" => "1:29456:2",
            "ts" => "04/22-17:47:29.436774"

}

I can see you are trying to chop up what I have labelled as field2, but your regular expression appears not to be valid so I cannot tell what you want there. In the dissect you can replace [%{field2}] with [%{field2}:%{field3}:%{field4}] if you want the three subfields split. And obviously you can call them anything that like.

If you do not want the contents of field1 then you can replace [%{field1}] with [%{}], or even if it is always empty.

[johndowe]

Thanks a lot badger for this... I have tried it out and it works, if I want to chop up further restofline field , so one ip as the source IP and the other as source IP is that possible?

[Badger]

Yes, you can use a second grok. For that ICMP message I think this would work

grok { match => { "restOfLine" => "^{(?<protocol>.*?)\}\s(?<source_ip>.*?)\s->\s(? 
<destination_ip>.*)" } }

Note that if you have other types of lines (other than ICMP) you can match restOfLine against an array of patterns. That's why I dealt with the first part of the line (which is constant across lines) in the first grok.

Note also that I anchored the ICMP pattern with ^, so that it fails quickly if it is not an ICMP line.