Grok Help

Elastic Stack Logstash
1

Hi

I'm new to ELK and Grok, so sorry if this seems to be basic, but I just can't seem to get this to work.

Basically I am trying to get a few more Cisco ASA logs parsed (others are essentially being passed as per http://www.gregmefford.com/blog/2014/09/24/analyzing-cisco-asa-firewall-logs-with-logstash/) . I'm hoping in getting the issue below resolved it will set me off on my way.

Basically it is around ASA-5-304001 which looks like the following (2 examples below)

172.29.251.22 Accessed URL 172.31.252.105:/hosting/discovery
10.10.29.102 Accessed URL 172.29.254.40:http://test.domain.local/office/cgi/cgiCAV.exeCN=TS01&IS=10.10.16.32

the grok I am looking to use and am having some success with is

%{IP:srcp_ip} Accessed URL ?(%{IP:dst_ip}:)?(%{URI:uri})

This doesn't match the first line , I think due to the : character (just before /hosting).
The second line only matches as far as the & symbol in the URI

I assume both of these characters a some sort of special characters?

Any help is appreciated and thanks in advance

Thanks

2

The reason that your pattern won't match the first line is the use of URI. Looking at its definition,

it only matches strings that start with letters followed by "://". Perhaps URIPATH would be a better choice when matching that kind of line? Or just GREEDYDATA if the paths/URLs are always the last thing on each line?

3

Hi

Thank you so much.

I had got it to "%{IP:srcp_ip} Accessed URL %{IP:dst_ip}:?(%{URI:uri}|%{URIPATH:uri})" with a bit of luck.

But the GREEDYDATA is a winner

Thanks

4

Can you show your pattern line for this paticular message ASA-5-304001 ...thanks

5

Hi

I can’t take credit for all of this

Here is the log stash input config

6

Hi Rhyse
Thanks for your kind reply , i could not find the config ... can you post it here

7

Hi Rhys
Did you attach the config.
Thanks

8

Sorry, it looks like it was stripped out of the email, see below

filter {

if [message] =~ "%ASA-" {

mutate { add_tag => ["cisco-asa"] }

# Split the syslog part and Cisco tag out of the message
grok {
  match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_message}"]
}

# Parse the syslog severity and facility
syslog_pri { }

# Parse the date from the "timestamp" field to the "@timestamp" field
date {
  match => ["timestamp",
    "MMM dd HH:mm:ss",
    "MMM  d HH:mm:ss",
    "MMM dd yyyy HH:mm:ss",
    "MMM  d yyyy HH:mm:ss"
  ]
  timezone => "Europe/London"
}

# Clean up redundant fields if parsing was successful
if "_grokparsefailure" not in [tags] {
  mutate {
    rename => ["cisco_message", "message"]
    remove_field => ["timestamp"]
  }
}


# Extract fields from the each of the detailed message types
# The patterns provided below are included in Logstash since 1.2.0
grok {
  match => [
    "message", "%{CISCOFW106001}",
    "message", "%{CISCOFW106006_106007_106010}",
    "message", "%{CISCOFW106014}",
    "message", "%{CISCOFW106015}",
    "message", "%{CISCOFW106021}",
    "message", "%{CISCOFW106023}",
    "message", "%{CISCOFW106100}",
    "message", "%{CISCOFW110002}",
    "message", "%{CISCOFW302010}",
    "message", "%{CISCOFW302013_302014_302015_302016}",
    "message", "%{CISCOFW302020_302021}",
    "message", "%{CISCOFW305011}",
    "message", "%{CISCOFW313001_313004_313008}",
    "message", "%{CISCOFW313005}",
    "message", "%{CISCOFW402117}",
    "message", "%{CISCOFW402119}",
    "message", "%{CISCOFW419001}",
    "message", "%{CISCOFW419002}",
    "message", "%{CISCOFW500004}",
    "message", "%{CISCOFW602303_602304}",
    "message", "%{CISCOFW710001_710002_710003_710005_710006}",
    "message", "%{CISCOFW713172}",
    #"message", "%{CISOCFW304001}",
    "message", "%{CISCOFW733100}"
  ]
}

grok {
  patterns_dir => ["/etc/logstash/patterns/cisco-asa"]
  match => [
     "message", "%{CISCOFW304001}",
     "message", "%{CISCOFW405001}",
     "message", "%{CISCOFW713903}"
  ]
 }


# Show Source IP locations
geoip {
    source => "src_ip"
    target =>  "geoip_src_ip"
    add_tag => ["geoip"]
}

geoip {
    source => "dst_ip"
    target =>  "geoip_dst_ip"
    add_tag => ["geoip"]
}

geoip {
    source => "ip"
    target =>  "geoip_ip"
    add_tag => ["geoip"]
}

}
}

My pattern file

# ASA-5-304001CISCOFW304001 %{IP:src_ip} Accessed URL %{IP:dst_ip}:%{GREEDYDATA:uri}
# ASA-4-405001
CISCOFW405001 Received ARP request collision from %{IP:src_ip}/%{MAC:src_mac} on interface %{DATA} with existing ARP entry %{IP:dst_ip}/%{MAC:dst_mac}
# ASA-4-713903CISCOFW713903 (Group = %{DATA:tunnel_group}, |)IP = %{IP:ip}, %{GREEDYDATA}

Sorry about the formatiing