The Elasticsearch Ingest Pipeline approach will be your best bet here. I expect the issue you linked to will be resolved by defining a default ingest pipeline.
The documentation can fill in the details but briefly you can process your RUM data using these steps:
Create a pipeline definition. This will put the GeoIP results under a top-level user.geo field:
PUT _ingest/pipeline/apm_user_geoip
{
"description": "Resolve GeoIP information for APM events",
"processors": [
{
"geoip": {
"field": "context.user.ip",
"target_field": "user.geo",
"ignore_missing": true
}
}
]
}
apm-server can register this for you if you prefer (see the docs).
Direct apm-server to use this pipeline when indexing, update apm-server.yml:
Events produced by the RUM (and other) agents will get the same treatment. Note that these fields are not indexed by default - you'll have to update your mapping manually to achieve that until https://github.com/elastic/apm-server/issues/1283 is resolved.
Thank you very much Gil for that information. I did all that you mentioned and was indeed getting the correct pipeline result of converting context IP but missed the need to map the resulting lon-lat into a geo_point. That brought me to finding the appropriate reference and we will be testing this today.
Kind regards,
Ronald
PS There is a reference to the geo_point mapping in 6.x but not in 6.6/6.5/lower. Is this new?
However, I am encountering a bit of a problem.
What I did was create a definition.json for the pipeline and configure apm-server.yml so that registration with Elasticsearch and APM is automated. Without creating a mapping, this results in a float for the lon-lat parameters, and I can't change the mapping anymore when the apm index has been created.
What I was able to do was create a geo_point mapping and inject it to the apm index BEFORE enabling the APM-Server but this results in the creation of new fields like context.service.name.keyword once the APM-Server onboards which then messes with all the Kibana UI for searching, like I can't see the APM services because the service names get into context.service.name.keyword instead of context.service.name (please see the filter in the screenshot)
As I am new to Painless and pipeline definition writing, I couldn't figure a way to force the geo_point type for location in the definition.json file. I've tried also to look for clear documentation on this but couldn't find any.
I'm know I'm doing something wrong here. I know I can reindex the default apm index to a new one so that the location field gets converted to a geopoint but I don't think that is correct either as there shouldn't be a need to do any manual task after correct configuration and setup is done.
I would greatly appreciate a point in the right direction for this please.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.