Data for browser, location is not populating

[sameer7 (sameer)]

Hi, we've an integration with apm rum js agent and are tracking metrics with User Experience dashboard. But it seems the data for browser, location, is not getting populated. Not sure what we're missing here.

We do see other metrics such as page load times, errors but these details don't seem to populate. In the server yaml file we have used the default flag for capture_personal_data.

[emilioalvap (Emilio Alvarez Piñeiro)]

Hi @sameer7,

Can you please start by checking whether the data is available on your APM data by running the next query?

GET traces-apm*,apm-*,logs-apm*,apm-*,metrics-apm*,apm-*/_search
{
  "query": {
    "bool": {
      "filter": []
    }
  },
  "size": 0,
  "aggs": {
    "values": {
      "terms": {
        "field": "client.geo.country_iso_code",
        "size": 10
      }
    }
  }
}

You should see some results come back.

[sameer7 (sameer)]

Hi @emilioalvap, I don't get any data from this, this is the response

{
  "took" : 82,
  "timed_out" : false,
  "_shards" : {
    "total" : 125,
    "successful" : 125,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "values" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ ]
    }
  }
}

how should we proceed here?

[sameer7 (sameer)]

@emilioalvap can you please help here? Is it due to the ingest pipelines not working?

[emilioalvap (Emilio Alvarez Piñeiro)]

Hi @sameer7,

I would start by making sure the required data is where it needs to be. Can you check if the field client.ip is set for documents that are being ingested in apm-* indices? It should look something like this:

If client.ip is set, I'd recommend checking the setup for APM ingest pipeline, making sure that traces-apm.rum-<version> contains the following processor and that no custom pipeline is overriding it:

{
    "geoip": {
      "database_file": "GeoLite2-City.mmdb",
      "field": "client.ip",
      "ignore_missing": true,
      "on_failure": [
        {
          "remove": {
            "field": "client.ip",
            "ignore_failure": true,
            "ignore_missing": true
          }
        }
      ],
      "target_field": "client.geo"
    }
  },

You can also check geoip database stats by running the following query on Kibana Dev Tools. These stats should not contain any errors:

GET _ingest/geoip/stats

[sameer7 (sameer)]

Hi @emilioalvap i can see client.ip is set. And this is the config in ingestion.json

{
    "id": "apm_user_geo",
    "body": {
      "description": "Add user geo information for APM events",
      "processors": [
        {
          "geoip": {
            "database_file": "GeoLite2-City.mmdb",
            "field": "client.ip",
            "target_field": "client.geo",
            "ignore_missing": true,
            "on_failure": [
              {
                "remove": {
                  "field": "client.ip",
                  "ignore_missing": true,
                  "ignore_failure": true
                }
              }
            ]
          }
        }
      ]
    }
  },

Also we're not directly ingesting data in elasticsearch, we first do it in kibana and then in elasticsearch is this the reason the default pipeline are not working?

output.elasticsearch is set to false to facilitate the same.

I see that it's mentioned here to enable it: Parse data using ingest node pipelines | APM Server Reference [7.13] | Elastic