GeoIP lat lon fields are always number inspite of mapping it to float

Elastic Stack Elasticsearch
1

My Mapping is as follows:
{
"mapping": {
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"clientip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"device": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"geoip": {
"properties": {
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "geo_point"
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"loglevel": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"module": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"pid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"syslog_message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

My filter settings:
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:device} %{DATA:module}(?:[%{POSINT:pid}])?: %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}
grok {
match => { "message" => [
"[%{LOGLEVEL:loglevel}%{SPACE}] %{GREEDYDATA:syslog_message}",
"level=%{WORD:loglevel} %{GREEDYDATA:syslog_message}",
"%{IP:clientip}",
"%{GREEDYDATA:syslog_message}"
]}
}
geoip {
source => "clientip"

database => "/etc/logstash/GeoLiteCity.data"

target => "geoip"
add_tag => ["geoip"]
add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]

}
mutate {
convert => ["[geoip][coordinates]","float"]
}
}

Because lat,lon is always numbers the plotting on the map is in wrong position.
Please help resolve this.

Many Thanks!

2

Your mapping has

which is the recommended way to map geo coordinates. The logstash geoip plugin will create a location field, that is correctly formatted for a geo_point field.

I suggest you keep the geoip filter configuration simple, and don't try to override the coordinates from longitude and latitude. If needed, you could rename location to coordinates instead, or simply change your queries / visualisations to use the location field.

3

Thanks for the response!
I tried with the filter:
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:device} %{DATA:module}(?:[%{POSINT:pid}])?: %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}
grok {
match => { "message" => [
"[%{LOGLEVEL:loglevel}%{SPACE}] %{GREEDYDATA:syslog_message}",
"level=%{WORD:loglevel} %{GREEDYDATA:syslog_message}",
"%{IP:clientip}",
"%{GREEDYDATA:syslog_message}"
]}
}
geoip {
source => "clientip"
}
}

Issue is that the lat/lon fields are always numbers and not floats. For example an ip of 14.143.35.10 results in lat:20, lon:77 whereas the actual values should have been lat:20.9833, lon:77.5833.

Please help.

4

Any other suggestions to try out?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.