"_geoip_lookup_failure" own src/dst ip mapping don't work

Elastic Stack Logstash
1

Hi all.

Can someone help me to findout why I get a "_geoip_lookup_failure".
I have separate elasticsearch and logstash servers.
On elasticsearch I have install the plugin "ingest-geoip"
$ sudo bin/elasticsearch-plugin install ingest-geoip
On logstash I have install the plugin "logstash-filter-geoip"
$ sudo bin/logstash-plugin install logstash-filter-geoip

I have a complex grok filter to parse a proprietary firewall syslog stream.
A part of this filter is "geoip".
I thought I have only to map my index pattern to the geoip configuration option.

Thats my config file
#############################################################
input {
udp {
port => 514
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "<%{WORD}>%{TIMESTAMP_ISO8601:Syslog-Zeit}%{SPACE}%{IPV4:Src-IP}...<removed>...%{IPV4:Dst-IP}...<removed>"}
}

geoip {
  source => "Src-IP"
  target => "Dst-IP"
}

}
}

output {
elasticsearch {
hosts => ["<removed>"]
user => "<removed>"
password => "<removed>"
index => "syslog-%{+YYYY.MM}"
}
stdout { codec => rubydebug }
}
#############################################################

Logstash is loading the geoip database:
#############################################################
[2018-03-25T18:20:35,265][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
#############################################################

Thats my output with the "_geoip_lookup_failure"
#############################################################
{
"host" => "10.16.40.3",
"Dst-IP" => "8.8.8.8",
"Dst-Service" => "domain",
"type" => "syslog",
"Dst-Inf" => "dhcp",
"@timestamp" => 2018-03-25T13:55:59.207Z,
"In-Inf" => "eth0",
"Dst-Port" => "53",
"Action" => "Allow",
"Protocoll" => "UDP",
"Src-IP" => "192.168.1.2",
"message" => "<14>2018-03-25T15:55:58+02:00 Test Test/box_Firewall_Activity: Info Test Allow: FWD|UDP|eth0|192.168.1.2|65463|00:0c:29:ed:4e:20|8.8.8.8|53|domain|dhcp|Inet|Normal Operation|10.15.40.206|8.8.8.8|0|1|0|0|0|0||||||\n",
"@version" => "1",
"Syslog-Zeit" => "2018-03-25T15:55:58+02:00",
"Src-Port" => "65463",
"Rule-Name" => "Inet",
"Src-MAC" => "00:0c:29:ed:4e:20",
"tags" => [
[0] "_geoip_lookup_failure"
],
"FW-Engine" => "FWD",
"Boxname" => "Test"
}
#############################################################

Where is my mistake?

Best regards
Ji Ona

2

The IP address in the Src-IP field belongs to a private network and can therefore not be looked up using geoip, which only works for public IPs. This is why you are getting a lookup failure. Your config also looks weird as it seems you want to overwrite the Dst-IP field.

3

OK.
I had modify the settings to:

geoip {
source => "Dst-IP"
}

Now I see the geoip informations

{
"Boxname" => "Test",
"@version" => "1",
"@timestamp" => 2018-03-25T16:52:20.981Z,
"Syslog-Zeit" => "2018-03-25T18:52:20+02:00",
"message" => "<14>2018-03-25T18:52:20+02:00 Test Test/box_Firewall_Activity: Info Test Remove: FWD|UDP|eth0|192.168.1.2|54676|00:0c:29:ed:4e:20|8.8.8.8|53|domain|dhcp|Inet|Balanced Session Idle Timeout|10.15.40.206|8.8.8.8|20|1|92|76|1|1||||||\n",
"host" => "10.16.40.3",
"Protocoll" => "UDP",
"In-Inf" => "eth0",
"Dst-IP" => "8.8.8.8",
"Action" => "Remove",
"type" => "syslog",
"Src-IP" => "192.168.1.2",
"Src-Port" => "54676",
"geoip" => {
"location" => {
"lat" => 37.751,
"lon" => -97.822
},
"latitude" => 37.751,
"country_code2" => "US",
"country_code3" => "US",
"continent_code" => "NA",
"ip" => "8.8.8.8",
"country_name" => "United States",
"longitude" => -97.822
},
"FW-Engine" => "FWD",
"Src-MAC" => "00:0c:29:ed:4e:20",
"Dst-Port" => "53",
"Dst-Service" => "domain",
"Dst-Inf" => "dhcp",
"Rule-Name" => "Inet"
}

At next I will modify my config file to distinguish in- and outbound traffic.

Best regards
Ji Ona

4

I stumble to distinguish for in- and outbound traffic.
I try to "mutate" with "replace" and "add_tag" but nothing will work right.
Thats my actual config file.

################################################
input {
udp {
port => 514
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "<%{WORD}>%{TIMESTAMP_ISO8601:Syslog-Time}%{SPACE}%{IPV4:Src-IP}...<removed>...%{IPV4:Dst-IP}...<removed>"}
}

if [Dst-IP] != "%{RFC1918}" and [Src-IP] != "%{RFC1918}" {
geoip {
source => "Src-IP"
add_tag => [ "WAN-in" ]
}
}
else if [Dst-IP] != "%{RFC1918}" and [Src-IP] == "%{RFC1918}" {
geoip {
source => "Dst-IP"
add_tag => [ "WAN-out" ]
}
}
else if [Src-IP] == "%{RFC1918}" and [Dst-IP] == "%{RFC1918}" {
mutate {
replace => { "[geoip][timezone]" => "City/GMT" }
add_tag => [ "LAN-LAN" ]
}
}
output {
elasticsearch {
hosts => ["<removed>"]
user => "<removed>"
password => "<removed>"
index => "syslog-%{+YYYY.MM}"
}
stdout { codec => rubydebug }
}
###############################################

Is the %{RFC1918} a valid default filter pattern?
Are my conditionals on the right position in the config file?
In my understanding I have can use "geoip" for public IPs and need the "mutate settings" for private IPs.

Kind regards
Ji Ona

1 Like
5

The way to do this is to combine the cidr filter and the geoip filter. Here are two bits of code based on my ElastiFlow solution that show how to do this:

First use cidr filter to match against private IP address ranges...

cidr {
  address => [ "%{[dst_addr]}" ]
  network => [ "0.0.0.0/32", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7", "127.0.0.0/8", "::1/128","169.254.0.0/16", "fe80::/10","224.0.0.0/4", "ff00::/8","255.255.255.255/32" ]
  add_field => { "[dst_locality]" => "private" }
}

If IP isn't private, lookup geo location...

if [dst_locality] != "private" {
  geoip {
    source => "[dst_addr]"
    target => "[geoip_dst]"
}
3 Likes
6

Hi Robert.
Thank you for your suggestion. Unfortunately I'm currently not an expert in logstash.
Could you please shortly explain your example for my requirements.
My grok filter is indexing the source end destination IPs to Src-IP and Dst-IP.
Depends on traffic flow (LAN to WAN or vice versa) the IPs can be private, public or both private/public.
Is it right if I modify your suggestion on that way:

cidr {
address => [ "%{[Dst-IP]}" ]
network => [ "0.0.0.0/32", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7", "127.0.0.0/8", "::1/128","169.254.0.0/16", "fe80::/10","224.0.0.0/4", "ff00::/8","255.255.255.255/32" ]
add_field => { "[dst_locality]" => "private" }
}

if [dst_locality] != "private" {
geoip {
source => "[Dst-IP]"
target => "[geoip_dst]"
}
}

Kind regards.
Ji Ona

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.