[GEOIP] Get target value as geoip from json input

[cs0sf]

Hi,

I am trying to make use of geoip plugin so that the IP field could be mapped to geo location, etc

My sample log looks like the following:
{"key1":"blah", "network":[{"IP":"80.80.229.213"},{"IP":"80.80.229.216"}]}

Sample logstash config:
input {
file {
path => ["/home/ras/Code/corp/elkstackv6/logstash-6.0.0/test.log"]
sincedb_path => '/tmp/sincedb.path'
start_position => beginning
codec => json { charset => "ISO-8859-1" }
}
}
filter {
geoip {
source => "[network][IP]"
target => "[network][GeoIP]"
}
}
output {
stdout { codec => rubydebug }
}

output:
{
"key1" => "blah",
"path" => "/home/code/elkstackv6/logstash-6.0.0/test.log",
"@timestamp" => 2017-11-27T10:23:37.848Z,
"@version" => "1",
"host" => "lab",
"network" => [
[0] {
"IP" => "80.80.80.9"
},
[1] {
"IP" => "80.80.60.60"
}
],
"tags" => [
[0] "_geoip_lookup_failure"
]
}

[guyboertje]

This is saying that "80.80.60.60" can't be found in the Maxmind GeoIP database.

[guyboertje]

FROM RIPE

inetnum:         80.80.60.0 - 80.80.60.255
netname:         PORTUS
descr:           H1Telekom d.d.
descr:           dial-up Osijek

[cs0sf]

I afraid but this is not the case.

i changed the log format to look what is causing this and it seems to be the access to 'IP' which is present in list of dictionary

if I change the above mentioned log sample with
{"key1":"blah", "IP":"80.80.60.60", "network":[{"IP":"52.33.146.13"},{"IP":"52.40.59.173"}]}

and conf with:
geoip {
source => "[IP]"
target => "[GeoIP]"
}

which gives valid output
{
"key1" => "blah",
"GeoIP" => {
"timezone" => "Europe/Zagreb",
"ip" => "80.80.60.60",
"latitude" => 45.1667,
"country_name" => "Croatia",
"country_code2" => "HR",
"continent_code" => "EU",
"country_code3" => "HR",
"location" => {
"lon" => 15.5,
"lat" => 45.1667
},
"longitude" => 15.5
},
"path" => "/home/code/elkstackv6/logstash-6.0.0/test.log",
"@timestamp" => 2017-11-27T17:04:15.069Z,
"IP" => "80.80.60.60",
"@version" => "1",
"host" => "xlabs",
"network" => [
[0] {
"IP" => "52.33.146.13"
},
[1] {
"IP" => "52.40.59.173"
}
]
}

If there any way by which we can have geo info for IP type data which is present in list of dictionary ? ( I think we are unable to access the IP inside list of dictionary and hence getting "_geoip_lookup_failure")

[guyboertje]

Sorry, you are correct.
The real problem though is that the value of the field [network] is an Array.

Q: Does it always have two entries only?

This means you need to need to use this nested field reference:
[network][0][IP] to pluck the first entry which is a Hash or Map of "IP" => "80.80.229.213"
[network][1][IP] to pluck the second entry which is a Hash or Map of "IP" => "80.80.229.216"

You could consider using the split filter which will create two new events (cloned from the original) each one having a different IP field.

If you decide to split then you will not need the [network][N][IP] field reference.

[cs0sf]

Q: Does it always have two entries only?
A: No, it can be of any length including 0

Yes, Split can be a good candidate to be used here but it will create cloned events which is somewhat i would keep distance from
Array is a complex data structure here and i think there is no solution to the problem except changing the way log looks

Thanks @guyboertje for finding time

[guyboertje]

You are welcome.