Syslog additional fields

Elastic Stack Logstash
1

Hi

We use a default filter for syslog messages, so they get parsed properly. However I wondered if it would be possible to match the message against a secondary pattern and extract new fields from the message:
Dec 4 14:12:46 10.1.1.254,149 338732: 338728: *Dec 4
13:13:38.097 GMT: %SEC-6-IPACCESSLOGP: list FILTER_INTERNET_IN denied
tcp 1.2.3.4(53261) -> 5.6.7.8(5000), 1 packet

In this case i would like to store the 1.2.34 in a new field like offending_ip

Thanks in advance

Kind regards

Theo

2

Absolutely you just have to write a grok for it. I did it for you just because I had a minute.

grok {   
match => { "message", "%{SYSLOGTIMESTAMP} %{IPV4},%{BASE10NUM} %{BASE10NUM}: {BASE10NUM}: \*%{SYSLOGTIMESTAMP} %{NOTSPACE} %{PROG}: %{WORD} %{PROG} %{WORD} %{WORD} %{IPV4:offending_ip}\(%{BASE10NUM}\) -> %{IPV4}\(%{BASE10NUM}\),%{GREEDYDATA}" }
}

That would actually parse the entire log you can take as much of it as you want. I dont know what your initial syslog part config looks like but if you post it I can show you how this would go in your config.

Jack West

3

You could also grok in multiple passes. If you have a basic syslog grok pattern, you could use conditionals and regular expressions to test for lines that have IPs in them, and then further grok those fields. You'd just have to use that field name in stead of "message" in subsequent grok blocks.

4

Guys

Thanks a lot, I will test today

Kind regards

Theo

5

hi i have been trying for quite some time but i cannot get it to work .... here is my config ..

file >>>>>>>> 02-beat-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
udp {
port => 5140
type => "syslog"
}

}
file >>>>>>>>>>>>>03-cisco-filter.conf

filter {
grok {
match => { "message" => "%{CISCOFW710001_710002_710003_710005_710006 %{WORD:protocol} (?:request|access) %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}" }
}
}
# Parse the syslog severity and facility
syslog_pri { }
dns {
reverse => [ "host" ]
action => "replace"
}
geoip {
source => "src_ip"
target => "geoip"
database => "/opt/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
# do GeoIP lookup for the ASN/ISP information.
geoip {
database => "/opt/logstash/GeoIPASNum.dat"
source => "src_ip"
}
}
~
file >>>>>>>>>>>>>>10-syslog-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}

}
file >>>>>>>>>>>>>> 30-elasticsearch-output.conf

output {
elasticsearch {
hosts => ["localhost:9200"]

sniffing => true

manage_template => false

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

document_type => "%{[@metadata][type]}"

}
stdout {
codec => rubydebug
}

}
.... i have been trying al sort of thing for quite some time but could not make it to work ...
I have several Cisco Devices which i want to send its log to this elk but i can't . i have this on my kibana which shows log is being shipped but i cant analyse base on source ip, port etc etc ...
message:<166>32926208: ha-ir1: Mar 28 23:50:24: %FMANFP-6-IPACCESSLOGP: SIP0: fman_fp_image: list INCOMING-FILTER denied udp 81.221.15.0(45498) GigabitEthernet0/0/0-> 202.134.25.220(53), 1 packet @version:1 @timestamp:March 28th 2017, 23:50:25.251 type:syslog host:10.254.36.190 tags:_grokparsefailure syslog_severity_code:5 syslog_facility_code:1 syslog_facility:user-level syslog_severity:notice _id:AVsUicjuYvvnRQWpC-6b _type:syslog _index:logstash-2017.0 .... appreciate someone would help out

6

@mhalatuituia, please start a new thread for your issue.

7

hi Jack
i comment on your post with my config and you suggest i should start a new thread. Please share your idea about my idea where i wrong as i really need this to work for my situation ... thanks in advance for your time and support

8

@Christian_Dahlqvist any comment on the new thread ....