RUM JS not collecting user agent location

marone · July 29, 2022, 2:26pm

This is my 3rd time configuring elastic APM, and always face a problem populating the "Page load distribution" over countries. By default elastic uses geoip processor to add client location for rum data but it's not the case for me. There is no index called traces-apm.rum-default created for RUM data.

Checking https://demo.elastic.co User experience dashboard indeed have the dashboard populated with data and the traces-apm.rum-default index exists.

I configured RUM for a Django application, I added to the base template the following:

<script src="{% static 'js/elastic-rum-based-umd-5.12.js' %}">

</script>
  <script type="text/javascript">
    elasticApm.init({
    serviceName: "frontend_dev_app",
    serverUrl: "http://localhost:8200",
    distributedTracingOrigins: ["http://localhost:8000", "http://127.0.0.1:8000", "http://localhost:8200"],
    pageLoadTraceId: "{{ apm.trace_id }}",
    pageLoadSpanId: "{{ apm.span_id }}",
    pageLoadSampled: {{ apm.is_sampled_js }}
  })
// some code...
    </script>

I am using elastic 7.15.
What am I missing? I checked the doc for rum many times but could not find how to include user agent location for rum data, and could not figure out why I don't have an index like traces-apm.rum-default?

The error I'm having:

Thanks.

Alberto_Delgado · August 2, 2022, 3:16pm

Hi @marone,

Thanks for reaching out and apologies for the delay.

RUM doesn't handle anything related to ip address and geo information. That is something that APM server handles. That's why you were not finding information on the RUM documentation.

--

Please, being in the devtools console page execute the following query:

GET /_ingest/geoip/stats

If the output you are getting is something like this:

Screenshot 2022-08-02 at 16.54.06

It means that there has been some issue while downloading the geoip database. So, what you need to do is to "turn it off and on again" the geo pipeline which will force the download of the geoip database

How to turn it off:

PUT /_cluster/settings?pretty
{
  "persistent" : {
    "ingest.geoip.downloader.enabled" : false
  }
}

How to turn it on:

PUT /_cluster/settings?pretty
{
  "persistent" : {
    "ingest.geoip.downloader.enabled" : true
  }
}

After turning it on you will see Elasticsearch logs like this ones:

If you execute the query GET /_ingest/geoip/stats again you will see that now the database has been downloaded properly:

After doing this, if you want to get rid of the "unable to perform term join" issue, what you need to do is to visit your webpage to generate new events, then the User Experience Page will show the data properly, for instance:

Thanks,
Alberto

marone · August 2, 2022, 3:38pm

Thank you so much @Alberto_Delgado for the precise answer. When doing the steps you described indeed I didn't find the geoip database, once I turn it off/on it is downloaded, and didn't get any geo data after, I believe because I tested it on localhost right? I need to deploy it ASAP and get back toyou if needed .

Marwane.

marone · August 25, 2022, 10:00am

Hey again @Alberto_Delgado,
First sorry for the long time it took. I deployed an application internally in our servers and tried to access it via the public url but still the map empty I checked also with GET /_ingest/geoip/stats to make sure that there are some database, and did the steps again but still have the error message unable to perform term join. What could be wrong please?

kyungeunni · August 26, 2022, 3:42pm

Hi @marone,

I'm so sorry that the issue is not solved.
At this point we should check if the relevant fields are generated in the documents. Could you please run this query in Dev Tools:

GET apm*/_search
{
  "query": {
      "bool": {
        "filter": [
          {
            "exists": {
              "field": "transaction.experience"
            }
          },
          {
            "exists": {
              "field": "client.geo.country_iso_code"
            }
          }
        ]
      }
    },
  "_source": ["client.geo.country_iso_code"]
}

This query is to see if the User Experience documents contain the geo location data.

Can you see any hits (hits.total.value > 0)? If you can, data required for the Page load distribution map are generated by the geoip processor correctly and we need to investigate this issue further.
If you don't see any hits with the above query, we can try to debug by setting a custom header X-Forwarded-For(it is useful if you are running things locally to debug). When initiating apm, try adding this option:

elasticApm.init({
    ....
    apmRequest: ({ xhr }) => {
      xhr.setRequestHeader('X-Forwarded-For', <ip-address-to-be-looked-up>)
      return true;
    }
})

In this case, the ingest process should add client.geo.country_iso_code after looking up the location of IP when storing the documents. Try visiting your website and running the first query again, you should get hits. Then the User Experience Page should show the data properly.

If any of these doesn't work out, please let us know Happy to investigate it further.

Thanks,
Kyungeun

marone · August 30, 2022, 1:19pm

Thank you @kyungeunni for your time. I tried the above code and still no client.geo.country_iso_code field in the apm* index, this is what I did:

retrieved my public ipv4 using https://whatismyipaddress.com/
updated the rum code as follow:

import { init as initApm } from '@elastic/apm-rum'
var apm = initApm({
  // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
  serviceName: 'carfront',
  // Set the version of your application
  // Used on the APM Server to find the right sourcemap
  serviceVersion: '0.90',
  environment: 'demo',
  secretToken: '',
  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: 'http://localhost:8200',
  // For distributed tracing to different origin (CORS)
  distributedTracingOrigins: ['http://cardatabase:8080', 'http://localhost:8080'],
  apmRequest: ({ xhr }) => {
   xhr.setRequestHeader('X-Forwarded-For', 'xxx.xxx.xxx.xxx')
   return true;
 }
})

export default apm;

started the server but showing no errors when opening chrome dev tools, the request made by apm has no header 'X-Forwarded-For':

image876×659 36.2 KB

What I did is it correct? Thanks.

kyungeunni · August 30, 2022, 2:26pm

Hi @marone,

Yes, you should see the X-forwared-header in the request header if configured correctly:

Your configuration looks correct (apart from the missing , right before apmRequest: ({ xhr }) => { line, but I'm sure it's a typo).

I'm wondering if it's cache related issue(either the requests or the javascript file). Could you try again after disabling cache on your browser?

stephenb · August 30, 2022, 2:37pm

Hi @kyungeunni
Question: Are you testing this only on internal browsers / IP addresses if so there will no geo ip information as internal IP address can not be looked up / converted to geo data.

kyungeunni · August 30, 2022, 2:41pm

Hi @stephenb

No, I'm testing this with public IPs, also @marone seems to be on the same page:

retrieved my public ipv4 using https://whatismyipaddress.com/

marone · August 30, 2022, 2:43pm

Yes was a typo only when I did copy/paste here when answering you.

I disabled cache, tried incognito mode and changed the web browser to Firefox and still not working.

marone · August 30, 2022, 2:48pm

One more detail, the application is running using docker on local on windows. is it related maybe?

stephenb · August 30, 2022, 2:50pm

But I see all this

serverUrl: 'http://localhost:8200',

That tells me your browser and the APM server are running on the same host... so that would not use your public IP. Do you see your public IP in the RUM data?

Just double checking... I have seen people spend a lot of time on very similar

marone · August 30, 2022, 2:52pm

Hey @stephenb, I've added the above code and 'xxx.xxx.xxx.xxx' is my public ip to debug

kyungeunni · August 30, 2022, 3:05pm

That tells me your browser and the APM server are running on the same host...

Yes, that's why I've suggested setting X-Forwarded-For. When APM Server receives the request and X-Forwarded-For header exists, its value will be used to determine the client's GEO location, according to here: Anonymous authentication | APM User Guide [8.11] | Elastic

What I'm doing here is basically taking advantage of this behaviour of APM server, I'm providing the public IP to X-Forwarded-For header so that APM server will look up the geo location of this public IP, rather than my internal IP, if that makes sense.(It is a hack to test geo ingest pipeline locally.)

I'll attach some screenshots from my local tests:

Request sent:

image778×1164 111 KB
RUM document from the request:

image854×704 48.5 KB

stephenb · August 30, 2022, 3:09pm

Can you show a sample document from the RUM Agent?

Are you seeing your public ip in the client.ip field

simitt · August 31, 2022, 11:01am

@marone please also enable DEBUG logging (logging.level: debug) in the apm-server for a short time. When enabled, it should log the request details from the RUM request.

Please share the debug logs for a specific RUM request (and strip away any sensitive information first). It will be especially interesting to see if a client.ip is set at this point.

marone · August 31, 2022, 2:21pm

Hello and thanks @all for your support.

here is a sample document of RUM data:

The problem is that the IP is private, and not taking the public ip provided manually using the X-Forwarded-For header.

I added the logging.level: debug to apm server and following the logs when refreshing the carfront page: apm server logs - JustPaste.it (could not paste it here because they are too long)

FYI, I'm using ELK and apm server version 7.15.2

marone · September 19, 2022, 1:41pm

Hello again @kyungeunni, @all

For some reason after a while it worked I can see the geo location of the value we used with X-Forwarded-For when initializing apm agent:

My question now is: How can I pass this header dynamically to elastic for all users visiting the webpage? when I did some research on google, found that they use Nginx and they set this header for incoming http request. is it the only way to do it dynamically please?

Thank you.

Alberto_Delgado · September 20, 2022, 1:58pm

Hi @marone,

Glad that worked!

when I did some research on google, found that they use Nginx and they set this header for incoming http request.

Bear in mind this description from MDN about X-Forwarded-For

"The X-Forwarded-For (XFF) header is a de-facto standard header for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or a load balancer. When traffic is intercepted between clients and servers, server access logs contain the IP address of the proxy or load balancer only. To see the original IP address of the client, the X-Forwarded-For request header is used."

You need to configure your environment to handle the scenario described above, you cannot do such logic from the client side. (excluding the local modification for testing that you made)

Apart from that, once you start having visitors you will start seeing their regions properly differentiated. Is the website already accessible over the internet?

Please, have a look at this link since might be helpful for you, too.

Thanks,
Alberto

marone · September 20, 2022, 2:28pm

Thanks a lot @Alberto_Delgado, and the website will be accessible only internally (using vpn), and I think I should set capture_personal_data to false from what I understood. Right?