How to adjust apm client .setLabel() index field type to "geo_point"

[rgroleau (Rejean Groleau)]

I am using the elastic apm client for nodejs to set custom labels when my IoT devices communicate with my IoT API. Here is the function documentation.

This API works great, but always sets the label value type to "string" (as mentioned in the docs).

I happen to be storing the geohash of my IoT devices GPS coordinates into a label called location, so I can see in visualizations labels.location of type string, with proper values.

But I cannot find an easy way of changing the apm-* index pattern field type for labels.location to anything other than string/number/etc. Basically geo_point is just not listed in the combo box list of supported index pattern field type.

Note that other fields in the apm-* index (such as client.geo.location) are listed in the index management page with the proper geo_point field type. I just cannot select that type when trying to change my own labels type from string (to geo_point).

Should I use a scripted field to convert my geohash string into the official "geo_point" type?

Or should I rather use a reindexing request to force my labels.location field into this non-listed geo_point format?

Or is there another apm client function that I can use to provide the device own self-known GPS localization info in geo_point format to the ES stack? Like, apm.setGeoLocation() or anything else than labels?

What is the official or recommended way of doing this?
Thanks!

[axw (Andrew Wilkins)]

@rgroleau welcome to the forum!

What you're recording sounds to me like it fits with the definition of client.geo.location in ECS (Elastic Common Schema). I think it would be sensible for the Elastic APM agents to provide a more specific API to record this information, and then we could record it with the correct field name and type – something like the apm.setGeoLocation() you propose.

Would that work for you? If so, I'll open a feature request to track it.

In the mean time you can use an Ingest node pipeline to rename labels.location to client.geo.location:

1. Create a "rename_location" pipeline:

PUT /_ingest/pipeline/rename_location
{
  "description" : "rename labels.location to client.geo.location",
  "processors" : [
    {
      "rename" : {
        "field": "labels.location",
        "target_field": "client.geo.location",
        "ignore_missing": true
      }
    }
  ]
}

2. Query the existing "apm" pipeline:

GET /_ingest/pipeline/apm

Elasticsearch responds with something like::

{
  "apm" : {
    "description" : "Default enrichment for APM events",
    "processors" : [
      {
        "pipeline" : {
          "name" : "apm_user_agent"
        }
      },
      {
        "pipeline" : {
          "name" : "apm_user_geo"
        }
      },
      {
        "pipeline" : {
          "name" : "apm_ingest_timestamp"
        }
      }
    ]
  }
}

3. Update "apm" pipeline to call the new "rename_locations" pipeline:

PUT /_ingest/pipeline/apm
{
  "description": "Default enrichment for APM events",
  "processors": [
    {
      "pipeline": {
        "name": "apm_user_agent"
      }
    },
    {
      "pipeline": {
        "name": "apm_user_geo"
      }
    },
    {
      "pipeline": {
        "name": "apm_ingest_timestamp"
      }
    },
    {
      "pipeline": {
        "name": "rename_location"
      }
    }
  ]
}

[rgroleau (Rejean Groleau)]

Hi @axw, thanks for that quick reply! Yes, apm.setGeoLocation() would be a great feature (and I will most likely not be the only one to use it since this is really useful). If you can open that feature request ticket I would greatly appreciate it! In the meantime I will try exactly what you suggested (the ingest node pipeline), thanks again!

[axw (Andrew Wilkins)]

@rgroleau FYI, I've opened the feature request: https://github.com/elastic/apm/issues/250

[rgroleau (Rejean Groleau)]

Thanks! I will be watching that!

[anjana1 (Anjana Asok)]

Hi I am stuck with the same problem too. SO after ingesting into pipeline , will we be able to use client.geo .location in Maps? and will that hold the value of my label?

[rgroleau (Rejean Groleau)]

Yes @anjana1, exactly. I did it and it works really well. It is much easier in geohash format though, since labels are strings and therefore cannot easily hold two floats as a string (lat+long) while a geohash is just that (a string with both values encoded as a text hash). I also used a conditional pipeline to avoid conflict when both X-Real-Ip is available (standard user geo ip pipeline) and my new labels.location (real device-side precise gps coordinates).

[anjana1 (Anjana Asok)]

@axw

GET /_ingest/pipeline/apm

I tried the same thing and my maps does not get any data . My label has a string value as this "45.567,-85.678".

[anjana1 (Anjana Asok)]

Also I did it using Developer tools and that's enough right? Or is it done via definition.json file in apm server.

[rgroleau (Rejean Groleau)]

Yes, the developer tools section/tab is how I did it too. Maybe your format is wrong? client.geo.location only supports a few formats, so you might have to script a little your "rename" pipeline to "split your string on the comma and put it as two fields lat/long". Not sure which pipeline type is the best for that (there are so many of them hehe) but take a look at the pipeline docs.

[anjana1 (Anjana Asok)]

I could successfully get the lat long on the maps . But now I want to show some kind of health data on these maps. eg:- Can I show the time taken by transactions on these map points so that we can drill down to the location where it has more transaction duration. I tried to us the transaction.duration.us on the filter but the graph does not show any data .Right now my graph is as below . But how can I show more data in this so that the map shows bigger circles for those duration that takes longer time

[anjana1 (Anjana Asok)]

were you also able to map other APM properties like transaction.duration.us . Like if you want to also show the transaction durations for that particular location. What would you do?

[rgroleau (Rejean Groleau)]

Personally I use the other "tab" in Kibana, the "Visualize" section (so I can put many of them in a nice "Dashboard" too). If you use a "Visualization" of type "Coordinate Map" you can plot the coordinates of your transactions (or anything else really) with "Metrics" in which you could use the minimum, maximum, average, sum or top hit of your transaction.duration.us value (so that the circles are a color, e.g. green or yellow when .us is low/fast, and another say red when the .us is high/slow). There are so many options! And don't forget the "Options" tab on the "Metrics", to change colors and everything: