Hello everyone,
I'm new to ELK. I'm learning to use it to store and analyze Suricata events. Everything is working except I can't get geoip working. The problem is the geoip source is from a sub-field and when I put the sub-field in my filter, it doesn't work. Here is an example of Suricata events.
{"timestamp":".....","in_iface":"br0","event_type":"http","src_ip":"10.200.238.66","src_port":88
45,"dest_ip":"10.200.237.111","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"example.com","url":"......","http_user_agent":"Mozilla","xff":"1.2.3.4","http_refer":"https://........","http_method":"GET","protocol":"HTTP/1.1","status":304,"length":0}}
Here is my filter.
if [http] {
mutate {
add_field => { "real_ip" => "%{[http][xff]}" }
}
geoip { source => "[http][xff]" }
}
As you can see, [http][xff] is the real client IP that Suricata writes to the event log based on the "X-Forwarded-For" header of the http request. I want geoip to look up for this IP address.
However, it seems like geoip cannot recognize this sub-filed. I always got "tags"=>["_geoip_lookup_failure"] and "geoip"=>{}. But the add_field => { "real_ip" => "%{[http][xff]}" } also use the sub-field and it works just fine.
I turned on logstash debug logging and I can see the config in the log as below.
[2016-11-23T00:08:42,954][DEBUG][logstash.filters.geoip ] config LogStash::Filters::GeoIP/@source = "[http][xff]"
[2016-11-23T00:08:43,368][DEBUG][logstash.filters.geoip ] config LogStash::Filters::GeoIP/@source = "src_ip"
[2016-11-23T00:08:43,373][DEBUG][logstash.filters.geoip ] config LogStash::Filters::GeoIP/@source = "dest_ip"
I don't quite understand why "src_ip" and "dest_ip" also appear in the log even though they are not specified as geoip sources in my config file.
The following in the log are obvious. GeoIP tried to look up the src_ip and dest_ip that are both internal IP addresses thus got "IP not found".
[2016-11-23T00:09:19,671][DEBUG][logstash.filters.geoip ] IP not found! {:exception=>com.maxmind.geoip2.exception.AddressNotFoundException: The address 10.200.238.66 is not in the database., :field=>"src_ip", :event=>2016-11-23T00:09:18.735Z}
[2016-11-23T00:09:19,672][DEBUG][logstash.filters.geoip ] IP 10.200.238.66 was not found in the database {:event=>2016-11-23T00:09:18.735Z }
[2016-11-23T00:09:19,672][DEBUG][logstash.filters.geoip ] IP not found! {:exception=>com.maxmind.geoip2.exception.AddressNotFoundException: The address 10.200.237.111 is not in the database., :field=>"dest_ip", :event=>2016-11-23T00:09:18.735Z }
[2016-11-23T00:09:19,673][DEBUG][logstash.filters.geoip ] IP 10.200.237.111 was not found in the database {:event=>2016-11-23T00:09:18.735Z }
Is there anything wrong with my geoip filter config? Is a sub-field supported as a geoip source?
Thanks. Any help is greatly appreciated!