I am currently mapping some of my clients coordinates that are reported on a nice map layer in Kibana. It looks good! But I would also like to be able to visualize these coordinates based on continent, country, regions, etc.
So here is my question: does anybody know of a simple way to convert geohash coordinates-only into a "richer" geolocation entry (with both the longitude/latitude coordinates and the extra infos such as continent, country or country code, region or city or others)?
If there is an existing ingest node pipeline that can do it that would be perfect, I am just too new to this to find it I guess. I am open to scripting it too if needs be, it's just that I thought I might not be the only one with that feature in mind.
Here is are some examples of what I have as a field on those APM transactions: client.geo.location: 9v32qxvx32f2 (example of a geohash) client.geo.location: { "lon": -97.0138, "lat": 32.4506 } (example of a geopoint)
And here is what I would like to "add" to it (much like the geoip ingest pipeline does): client.geo.continent_name: North America client.geo.country_iso_code: US client.geo.region_name: Texas client.geo.region_iso_code: US-TX client.geo.city_name: Grand Prairie
If the GeoLite2 City database is used, then the following fields may be added under the target_field : ip , country_iso_code , country_name , continent_name , region_iso_code , region_name , city_name , timezone , latitude , longitude and location . The fields actually added depend on what has been found and which properties were configured in properties .
Whenever the additional information is found, the attributes should be added by default.
I did not explain well enough my use case. I will try to put some emphasis on the section that I did not make clear enough, so I apologize in advance for the use of bold characters.
So here it is: I do not have an IP address. My clients are reporting through communications mechanisms that do not use IP, through IoT providers that all have the same range of IP addresses (in the same cloud servers), so IP addresses are useless to me. Converting those always yields the same coordinates. So my devices (my clients) are reporting through a real GPS antenna their real coordinates, very precise too.
So I am looking for means to convert only coordinates into continent/country/region/city,
and I am not looking to convert an IP address into coordinates+continent/country/region/city.
I am already using the geoip ingest pipeline on my web pages, and it works great. But this is not my use case for this question. For my IoT devices, that are reporting coordinates without an IP address, I don't see how I can use the geoip ingest pipeline to convert coordinates into continent/country/region/city, that pipeline only seems to support IP addresses as input.
Again, sorry for the bold characters, just trying to clarify what I was unable to do or to understand.
If you know of a way to convert "only coordinates" using any existing pipeline I would be glad to use it, maybe I just did not read all the proper documentation on all the existing pipelines (to my defense, there are a lot of them which I find absolutely awesome, I really like using Kibana, but it takes time to learn hehe).
This is the best I have found so far, but it seems like a lot of setup to get all proper fields (continent, country AND region AND city will make for a HUGE percolator from what I can guess): https://www.elastic.co/blog/using-percolator-geo-tagging/
This sounds like a use case for a geo_match enrich policy. You would need to populate an index that maps a geo_shape to the fields you want to include (continent, country, etc.)
When you're indexing, you would use an enrich processor that finds the geo_shape(s) that the coordinates fall within, and appends the fields from the matched enrichment document.
Hi @rgroleau. Thanks for sharing this question with us. I agree with Andrew that the geo_match enrich policy is the best option for this.
The Elastic Maps Service has country and some subdivision boundaries that you can use. For example, you can download the GeoJSON file from https://maps.elastic.co. Then use the GeoJSON Upload tool in Elastic Maps to upload the GeoJSON file into a new index containing the boundaries.
After the index has been created you can set up the geo_match enrich policy to use the new index and attach fields from the boundaries index to your documents at ingest.
@rgroleau We want to make this process much easier so we could really use some more information from you. Can you share what the lat/long coordinates represent? For example, are these the origins of a security threat, or building locations, or something else?
The more we understand the specific problems and datasets in play, the better we can set the priorities of how we build our solutions going forward.
The geo_match solution seems the best solution for now, thanks!
In our case the coordinates represent the IoT devices real precise GPS coordinates (using GPS/GLONASS antennas).
It really helps us right now since we have half a million devices all over the world and we can track any outage. Which is ok for a human looking at a map, but when automating the alerts (to say, detect if the west coast, or middle-west, or east-coast, or europe, or asia has any trouble right now) it is really nice to have the continent/country/regions/states/cities to be able to automate the decisions and alerts based on those. Typically when a provider goes down, a whole region is down, and alerts really helps with this (with region fields).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.