GeoIP lat,lon fields are numbers instead of float

Elastic Stack Logstash
1

Hello Team,

I see that the geoip.latitude, geoip.longitude are numbers and does not take decimal values, so a wrong position is plotted on the Tile Map.
For example an ip of 14.143.35.10 results in lat:20, lon:77 whereas the actual values should have been lat:12.9833, lon:77.5833.
I am not sure what is the mistake.
My filter config:
geoip {
source => "clientip"
}

GeoIP Index Mapping:
"geoip": {
"properties": {
"city_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "float"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
},
"postal_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},

Please help resolve this.

Many Thanks.

2

Is this your full filter config?

3

Please find the full filter Config below:
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:device} %{DATA:module}(?:[%{POSINT:pid}])?: %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}
grok {
match => { "message" => [
"[%{LOGLEVEL:loglevel}%{SPACE}] %{GREEDYDATA:syslog_message}",
"level=%{WORD:loglevel} %{GREEDYDATA:syslog_message}",
"%{IP:clientip}",
"%{GREEDYDATA:syslog_message}"
]}
}
geoip {
source => "clientip"
}
}

Thanks

4

Can you show me the raw output you are getting for all geoip fields please?

5

JSON document is as follows:
"_source": {
"module": "PublicIP",
"clientip": "14.143.35.10",
"geoip": {
"ip": "14.143.35.10",
"location": {
"lat": 20,
"lon": 77
},
"timezone": "Asia/Kolkata",
"latitude": 20,
"longitude": 77,
"country_code2": "IN",
"continent_code": "AS",
"country_code3": "IN",
"country_name": "India"
},
"timestamp": "Nov 16 11:30:21",
"message": "14.143.35.10",
"host": "14.143.35.10",
"@version": "1",
"@timestamp": "2018-11-16T11:30:23.638Z",
"device": "a8248fa5-28f0-4daf-9137-078f5ee9dc55"
},

Hope i have provided you the needed details.

6

I think your Longitude and Latitude fields should be set to "half_float" not "float" - try changing these in the template to see if that resolves.

7

@Eniqmatic, changing lat,lon to half_float does not solve the issue.
Anything else to try? Any help would be appreciated.
Please see template for GeoIP below:
"geoip": {
"properties": {
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "float"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},

8

Can you post the raw value now?

9

Its almost the same as earlier, please find below:
"geoip": {
"latitude": 20,
"country_code3": "IN",
"continent_code": "AS",
"timezone": "Asia/Kolkata",
"location": {
"lon": 77,
"lat": 20
},
"country_code2": "IN",
"longitude": 77,
"coordinates": [
77,
20
],
"country_name": "India",
"ip": "14.143.35.10"
},

10

What is your index name please? Can you do a GET using the development console on the index and post the mappings?

11

I ran the following and that is the value that seems to be the value that comes from the default geoip database:

input {
  generator {
    lines => ['14.143.35.10']
    count => 1
  } 
} 

filter {
  geoip {
    source => "message"
  }
}

output {
  stdout { codec => rubydebug }
}

and this results in:

{
  "@timestamp" => 2018-11-26T10:28:07.996Z,
  "message" => "14.143.35.10",
  "geoip" => {
    "ip" => "14.143.35.10",
    "longitude" => 77.0,
    "country_code2" => "IN",
    "timezone" => "Asia/Kolkata",
    "latitude" => 20.0,
    "continent_code" => "AS",
    "country_code3" => "IN",
    "country_name" => "India",
    "location" => {
      "lat" => 20.0,
      "lon" => 77.0
    }
  }
}
12

But the actual value for 14.143.35.10 should have been lat:12.9833, lon:77.5833 (https://ipinfo.io/).
When i checked the GeoLite2-City-Blocks-IPv4.csv, i dont find an entry for 14.143.35.10. So how does that work? Does it apply some rules based on available IPs and come up with some value?
Is there a way to change it?

13

But the actual value for 14.143.35.10 should have been lat:12.9833, lon:77.5833 (https://ipinfo.io/).
When i checked the GeoLite2-City-Blocks-IPv4.csv, i dont find an entry for 14.143.35.10. So how does that work? Does it apply some rules based on available IPs and come up with some value?
Is there a way to change it?

14

Can you do a get on the index as above?

15

Here is the index mapping output on dev console:
{
"logstash-2018.11.26": {
"mappings": {
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"clientip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"device": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"geoip": {
"properties": {
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "float"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"loglevel": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"module": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"pid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"syslog_message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}

16

That looks correct to me! Can you try another IP address instead?

17

The value you are getting seems to be what is in the bundled default database. If you need additional precision, it is possible to run it with non-default databases as well.

18

Just trying to understand, what happens if there is no match for the IP Address in the DB?
Other than maxmind DBs, are there anything else that is supported?

19

I have never used anything but the default, but would recommend checking out the docs. It seems like at least commercial MaxMind databases are supported, but am not sure if there are other free compatible options available.

20

Thanks!