GeoIP lat,lon fields are numbers instead of float


(prakash m) #1

Hello Team,

I see that the geoip.latitude, geoip.longitude are numbers and does not take decimal values, so a wrong position is plotted on the Tile Map.
For example an ip of 14.143.35.10 results in lat:20, lon:77 whereas the actual values should have been lat:12.9833, lon:77.5833.
I am not sure what is the mistake.
My filter config:
geoip {
source => "clientip"
}

GeoIP Index Mapping:
"geoip": {
"properties": {
"city_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "float"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
},
"postal_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},

Please help resolve this.

Many Thanks.


(Lewis Barclay) #2

Is this your full filter config?


(prakash m) #3

Please find the full filter Config below:
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:device} %{DATA:module}(?:[%{POSINT:pid}])?: %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}
grok {
match => { "message" => [
"[%{LOGLEVEL:loglevel}%{SPACE}] %{GREEDYDATA:syslog_message}",
"level=%{WORD:loglevel} %{GREEDYDATA:syslog_message}",
"%{IP:clientip}",
"%{GREEDYDATA:syslog_message}"
]}
}
geoip {
source => "clientip"
}
}

Thanks


(Lewis Barclay) #4

Can you show me the raw output you are getting for all geoip fields please?


(prakash m) #5

JSON document is as follows:
"_source": {
"module": "PublicIP",
"clientip": "14.143.35.10",
"geoip": {
"ip": "14.143.35.10",
"location": {
"lat": 20,
"lon": 77
},
"timezone": "Asia/Kolkata",
"latitude": 20,
"longitude": 77,
"country_code2": "IN",
"continent_code": "AS",
"country_code3": "IN",
"country_name": "India"
},
"timestamp": "Nov 16 11:30:21",
"message": "14.143.35.10",
"host": "14.143.35.10",
"@version": "1",
"@timestamp": "2018-11-16T11:30:23.638Z",
"device": "a8248fa5-28f0-4daf-9137-078f5ee9dc55"
},

Hope i have provided you the needed details.


(Lewis Barclay) #6

I think your Longitude and Latitude fields should be set to "half_float" not "float" - try changing these in the template to see if that resolves.


(prakash m) #7

@Eniqmatic, changing lat,lon to half_float does not solve the issue.
Anything else to try? Any help would be appreciated.
Please see template for GeoIP below:
"geoip": {
"properties": {
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "float"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},


(Lewis Barclay) #8

Can you post the raw value now?


(prakash m) #9

Its almost the same as earlier, please find below:
"geoip": {
"latitude": 20,
"country_code3": "IN",
"continent_code": "AS",
"timezone": "Asia/Kolkata",
"location": {
"lon": 77,
"lat": 20
},
"country_code2": "IN",
"longitude": 77,
"coordinates": [
77,
20
],
"country_name": "India",
"ip": "14.143.35.10"
},


(Lewis Barclay) #10

What is your index name please? Can you do a GET using the development console on the index and post the mappings?


(Christian Dahlqvist) #11

I ran the following and that is the value that seems to be the value that comes from the default geoip database:

input {
  generator {
    lines => ['14.143.35.10']
    count => 1
  } 
} 

filter {
  geoip {
    source => "message"
  }
}

output {
  stdout { codec => rubydebug }
}

and this results in:

{
  "@timestamp" => 2018-11-26T10:28:07.996Z,
  "message" => "14.143.35.10",
  "geoip" => {
    "ip" => "14.143.35.10",
    "longitude" => 77.0,
    "country_code2" => "IN",
    "timezone" => "Asia/Kolkata",
    "latitude" => 20.0,
    "continent_code" => "AS",
    "country_code3" => "IN",
    "country_name" => "India",
    "location" => {
      "lat" => 20.0,
      "lon" => 77.0
    }
  }
}

(prakash m) #12

But the actual value for 14.143.35.10 should have been lat:12.9833, lon:77.5833 (https://ipinfo.io/).
When i checked the GeoLite2-City-Blocks-IPv4.csv, i dont find an entry for 14.143.35.10. So how does that work? Does it apply some rules based on available IPs and come up with some value?
Is there a way to change it?


(prakash m) #13

But the actual value for 14.143.35.10 should have been lat:12.9833, lon:77.5833 (https://ipinfo.io/).
When i checked the GeoLite2-City-Blocks-IPv4.csv, i dont find an entry for 14.143.35.10. So how does that work? Does it apply some rules based on available IPs and come up with some value?
Is there a way to change it?


(Lewis Barclay) #14

Can you do a get on the index as above?


(prakash m) #15

Here is the index mapping output on dev console:
{
"logstash-2018.11.26": {
"mappings": {
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"clientip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"device": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"geoip": {
"properties": {
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "float"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"loglevel": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"module": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"pid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"syslog_message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}


(Lewis Barclay) #16

That looks correct to me! Can you try another IP address instead?


(Christian Dahlqvist) #17

The value you are getting seems to be what is in the bundled default database. If you need additional precision, it is possible to run it with non-default databases as well.


(prakash m) #18

Just trying to understand, what happens if there is no match for the IP Address in the DB?
Other than maxmind DBs, are there anything else that is supported?


(Christian Dahlqvist) #19

I have never used anything but the default, but would recommend checking out the docs. It seems like at least commercial MaxMind databases are supported, but am not sure if there are other free compatible options available.


(prakash m) #20

Thanks!