Hi,
I am trying to extract an IP address from a syslog message in log stash. When I run log stash in debugging mode to stdout using only this simple filter it maps to geoip correctly and the geoip plugin processes and outputs additional calculated fields:
filter {
grok {
match => { "message" => "%{IPV4:ip}"}
}
geoip {
source => "ip"
}
}
Input I am testing with:
Mar 12 04:05:30 ip-172-31-20-78 2018-03-12 message repeated 991 times: [04:05:30.109066 [!] Artillery has detected an attack from IP address: 114.115.146.121 for a connection on a honeypot port: 1433]
Mar 12 04:05:35 ip-172-31-20-78 2018-03-12 message repeated 1000 times: [04:05:35.252924 [!] Artillery has detected an attack from IP address: 114.115.146.121 for a connection on a honeypot port: 1433]
Mar 12 04:05:35 ip-172-31-20-78 2018-03-12 message repeated 1001 times: [04:05:35.692720 [!] Artillery has detected an attack from IP address: 114.115.146.121 for a connection on a honeypot port: 1433]
Mar 12 04:05:36 ip-172-31-20-78 2018-03-12 message repeated 1002 times: [04:05:36.270937 [!] Artillery has detected an attack from IP address: 114.115.146.121 for a connection on a honeypot port: 1433]
However when I adapt this pattern to my syslog logstash filter it is not returning the field or any of the latitude and longitude calculations like it does when using the simple filter. Here is the filter I am referencing:
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{IPV4:ip} %{GREEDYDATA:syslog_message} " }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
geoip {source => "ip" }
syslog_pri { }
date {match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]}
}
}
Now when I test the grok filter using the grok debugger it is indeed parsing out the ip field from the message as desired. Here is the output from the grok debugger:
{
"syslog_timestamp": [
[
"Mar 12 04:05:30"
]
],
"MONTH": [
[
"Mar"
]
],
"MONTHDAY": [
[
"12"
]
],
"TIME": [
[
"04:05:30"
]
],
"HOUR": [
[
"04"
]
],
"MINUTE": [
[
"05"
]
],
"SECOND": [
[
"30"
]
],
"syslog_hostname": [
[
"ip-172-31-20-78"
]
],
"IPORHOST": [
[
"ip-172-31-20-78"
]
],
"HOSTNAME": [
[
"ip-172-31-20-78"
]
],
"IP": [
[
null
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
null
]
],
"syslog_program": [
[
"2018-03-12 message repeated 991 times: [04:05:30.109066 [!] Artillery has detected an attack from IP address"
]
],
"syslog_pid": [
[
null
]
],
"ip": [
[
"114.115.146.121"
]
],
"syslog_message": [
[
"for a connection on a honeypot port:"
]
]
}
I am trying to determine why the parsing and assignment to the geoip field doesn't work in the updated logstash.conf file considering the more simplistic version is working correctly.
Thanks,
Chad