APM is working, but not User Experience

Hello,

I have a small problem with the "User Experienc" tab indeed, I set up APM on my Python/Flask application. APM works perfectly :

However, when I go to the "User Experience" tab I see nothing:

Also, in the "web app" dropdown I can't find the application which is however present in the "APM" tab. Is there a parameter that I have not defined?

Thanks in advance,
Best regards

Hello @A_Chichi, the User Experience app will show RUM (website) data collected from the RUM Agent. Have you instrumented your website with the RUM Agent?

If not, you can find more details on how to do that, here.

Thanks for your answer, you are right. I didn't have RUM.
Surprisingly I didn't find anything that redirected me to RUM when I searched the internet. RUM is a bit hidden.

It worked well by adding the necessary JS here

I have the same problem, even when using RUM.

After contacting Elastic Technical Support they claim that this is not supported, and redirected me here.

I do see the following error in my browser console when on the User Experience tab:

kbn-ui-shared-deps.js:375 TypeError: val.split is not a function
at apm.chunk.1.js:3
at kbn-ui-shared-deps.js:421
at kbn-ui-shared-deps.js:421
at li (kbn-ui-shared-deps.js:421)
at Nn.mapValues (kbn-ui-shared-deps.js:421)
at useUiFilters (apm.chunk.1.js:3)
at apm.chunk.1.js:3
at ua (kbn-ui-shared-deps.js:375)
at xs (kbn-ui-shared-deps.js:375)
at ml (kbn-ui-shared-deps.js:375)

Hi @Robbert-Jan, so you have the RUM Agent included on your website and you’ve configured this to beacon back to your APM Server, and when you go into the User Experience app, you see N/A for all the data?

Can you check that the date/time range you have selected in the top right, includes a period when you’ve had traffic to the website?

Also, if you go into the APM app (under Observability in the main menu), do you see the service you have set up for the RUM Agent, with data in there?

Hi @PaulB-Elastic ,
Yes that is correct, I can see the data in the APM tab, and there is a fancy button there telling me to visit the User Experience tab, but once I go there I don't see any data.

If I click on the JS button next to the service name in APM I see the following:
Service version

  • 22.02.2021

Agent name & version

rum-js 5.6.3

Edit: forgot to mention, that, yes, I did check the date/time range in the top right

Hey @Robbert-Jan , Can you please post screenshot of the User Experience tab, and what you see, and also any console logs from browser console, you can open that by pressing F12.
Regards

Hey @shahzad31 ,

here you go:

Yeah it seems like, there is no Web Application selected in the Web App Select Box.

Are you sure , you have RUM agent enabled, do you see that in APM services list? Also which version it is.

If you ES, you can debug the ES queries being executed by this page bu enabling this in your console

window.sessionStorage.setItem('apm_debug', true)
reload the page and all the queries will be logged in your kibana logs.

Also in the dev tools network tabs, you can see services API response, please check if it's returning your RUM Services list , see this example

@shahzad31
I am sure I have RUM agent enabled, I see it in the APM services list.
I also mentioned the version a few comments back: 5.6.3

Which page are you refering to here?

If you ES, you can debug the ES queries being executed by this page bu enabling this in your console
window.sessionStorage.setItem('apm_debug', true)
reload the page and all the queries will be logged in your kibana logs.

In the dev tools network tab I see that there is an empty array as response to the RUM Services list.
I also see an error as a response to the rumOverview request:

Request URL:

https://.eu-central-1.aws.cloud.es.io:9243/api/apm/ui_filters/local_filters/rumOverview?uiFilters=%7B%22environment%22%3A%22ENVIRONMENT_ALL%22%7D&filterNames=%5B%22location%22%2C%22device%22%2C%22os%22%2C%22browser%22%5D

  1. error: "Bad Request"
  2. message: "Invalid value undefined supplied to : strict_keys/0: { query: (({ filterNames: pipe(JSON, Array<"host" | "agentName" | "containerId" | "podName" | "transactionResult" | "serviceVersion" | "transactionUrl" | "browser" | "device" | "location" | "os" | "serviceName">) } & { uiFilters: string } & { start: DateAsString, end: DateAsString }) & { }) }/query: (({ filterNames: pipe(JSON, Array<"host" | "agentName" | "containerId" | "podName" | "transactionResult" | "serviceVersion" | "transactionUrl" | "browser" | "device" | "location" | "os" | "serviceName">) } & { uiFilters: string } & { start: DateAsString, end: DateAsString }) & { })/0: ({ filterNames: pipe(JSON, Array<"host" | "agentName" | "containerId" | "podName" | "transactionResult" | "serviceVersion" | "transactionUrl" | "browser" | "device" | "location" | "os" | "serviceName">) } & { uiFilters: string } & { start: DateAsString, end: DateAsString })/2: { start: DateAsString, end: DateAsString }/start: DateAsString"
  3. statusCode: 400

That error is bad but it won't break your page. But real reason why you are not getting any view is because of that empty array as response to RUM Services list.

I am talking about the User Experience app page for debug option..

@shahzad31
I've executed window.sessionStorage.setItem('apm_debug', true) in my browser, but I don't know where to find the Kibana logs.

I've checked the Logs and Metrics page in Elastic Cloud, but I don't see any queries being logged there.
I also don't see any queries shown in the Logs tab in Kibana: that just shows the logs in my filebeat index.

Do you have any idea how I can fix Kibana so that it detects that I do have RUM services data available?

@Robbert-Jan, this can come up pre-APM-6.4.0 if your apm-server.yml doesn't have indices configured (esp. apparently from docker images)

Edit: Second, if you're using Cloud, you could consider using an APM Cloud server. It helps simplify your setup. Here's a walkthrough where serverURL directly points to the APM Cloud instance.

At least from that last linked base example (v. 4.0.1) pointing to Cloud (ES, K, APM v7.11.1), I can get the base page to show but not errors on the UX bc of not having indices configured:

  1. forward error to APM server via RUM-JS, accepted per HTTP 202 response

  1. however index setup is only correct for apm-onboarding & not apm-error

  1. User Experience shows NULL data for last 30m where only error occurred bc on invalid index (doesn't show in screenshot, woops, but last box of screen of "errors" is NULL)

  1. However expanding timeframe to include the base page load (on correctly formatted apm-onboarding index), then the page loads

Edit: clarified #3

I'm going to leave, but recant that last part a little bit: the Kibana UI was failing to show data when I didn't have my RUM-JS agent properly tagging (screenshots were a sequential hodge-podge, sorry). Once I fixed that, though I don't get errors to show at the bottom of the User Experience, I do get all transactions to show in the UI. Again - w/APM Cloud server & not self-hosted, so the backend apm-server.yml may be accounting for the previous setup needs (not sure).

Hi @Stef_Nestor ,

Thanks for looking into this issue!

I'm using APM server image docker.elastic.co/apm/apm-server:7.10.0 with these indices configured:
output.elasticsearch:

  indices:

    - index: "apm-%{[beat.version]}-sourcemap"

      when.contains:

        processor.event: "sourcemap"

    - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"

      when.contains:

        processor.event: "error"

    - index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"

      when.contains:

        processor.event: "transaction"

    - index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"

      when.contains:

        processor.event: "span"

I've now updated these indices to include the metric and onboarding indices, exactly like your linked post. Unfortunately, this does not solve the problem.

I've also tried using the cloud hosted APM server, but that, also, does not solve the problem.

Is there anything else I can try?

@Robbert-Jan

This is the demo query UX app uses to get list of services, you can try running this in your kibana dev tools, and see what list of services you get, if you get nothing, trying removing filters like observer.version_major or agent.name. etc and check at what point it returns some services.
Also in this i used last 24h time range, you can also change that.

That should give you some idea.

GET traces-apm*,apm-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-24h",
              "lte": "now",
              "format": "epoch_millis"
            }
          }
        },
        {
          "term": {
            "transaction.type": "page-load"
          }
        },
        {
          "terms": {
            "agent.name": [
              "js-base",
              "rum-js",
              "opentelemetry/webjs"
            ]
          }
        },
        {
          "terms": {
            "processor.event": [
              "transaction"
            ]
          }
        },
        {
          "range": {
            "observer.version_major": {
              "gte": 7
            }
          }
        }
      ]
    }
  },
  "size": 0,
  "aggs": {
    "services": {
      "terms": {
        "field": "service.name",
        "size": 1000
      }
    }
  }
}

@shahzad31 thanks for your suggestion.

I executed the query as-is, and got the following (I redacted the actual service names):
{
"took" : 139,
"timed_out" : false,
"_shards" : {
"total" : 88,
"successful" : 88,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 9986,
"relation" : "eq"
},
"max_score" : null,
"hits" :
},
"aggregations" : {
"services" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "service1",
"doc_count" : 7377
},
{
"key" : "service2",
"doc_count" : 2496
},
{
"key" : "service3",
"doc_count" : 113
}
]
}
}
}

@shahzad31 , @Stef_Nestor Thinks that it could be that somehow, the necessary sub-objects _source.transaction.marks["agent", "navigationTiming"] aren't being sent by the RUM agent, or the APM server is not storing them.
In either case, they are not present in the output of

  1. GET apm-7.11.1-transaction-*/_search
    {"size": 2,"query": {"range": {"@timestamp": {"gte": "now-30m","lte": "now"}}}}

Does that help you pinpoint the problem?

I think it's really hard at this point to figure out the problem without support. Even if those fields aren't present , you should still see the service and page should load.
Maybe something weird is going on with this particular environment.
It might be helpful to see some sample documents from the run service you have.

Can you also share what you see in the service overview page for the rum service in the APM UI.