Identify unlabeled RUM activity in timeline

[stitch-bb8]

Kibana version:

Elasticsearch version: 7.5.2

APM Server version: 7.5.2

APM Agent language and version: Java 1.15.0

Browser version: Cordova browser, but essentially Android 8.1.0 Chrome Mobile WebView

Original install method (e.g. download page, yum, deb, from source, etc.) and version: not sure

Fresh install or upgraded from other version?: fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.**:

For our Javascript (Angular) RUM , we are using our own transactions, like:
Globals.APM.startTransaction(name, type, {managed: true});

And the Javascript/Angular RUM Agent is configured as:

            serviceName: 'twms-mobile',
            serverUrl: <our server URL>,
            environment: <our environment name>,
            disableInstrumentations: ['page-load', 'custom', 'user-interaction']
        });

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Please see attached screenshot. As you can see we are correctly getting distributed transactions and spans for a lot of stuff. I need help identifying what is going on during the part I annotated with the red line and the orange line. I believe the orange line is mostly network, but I'm not seeing anything that details SSL negotiation and such. Is the blue bar for the "POST" span include or exclude opening connections?

[Hamidreza (Hamidreza Jahtalabziabari)]

Hi @stitch-bb8,

Thanks for reaching out.

The blue bar for POST measures the time from when the XMLHttpRequest.send is called until the request's ready state changes to DONE. The space that is indicated by the orange line could be processing/parsing that is done on the server side before the Java agent captures the transaction (e.g. a middleware).

Regarding the red line, it depends on the application. for example if you're using angular's http service then there might be some processing that happens in the framework. The best way to debug these would be to use performance.mark and performance.measure (or create custom spans) in places that could introduce this delay.

Let me know if you have any other questions.

Cheers,
Hamid

[stitch-bb8]

Thanks, Hamid.
I'm still processing your response, but I am wondering if Breakdown Metrics might provide some of the answers I am looking for? Based upon the description at this URL, I think they would help me a lot because they seem to mirror some of what Chrome DevTool's "Timeline" view shows:

https://www.elastic.co/guide/en/apm/agent/rum-js/current/breakdown-metrics-docs.html

Network Analysis Reference  |  Tools for Web Developers

Going on the assumption that the breakdown metrics would help, I've tried to enable them but I didn't have any luck. Are custom transactions allowed to use breakdown metrics as long as they are configured as "managed"? If so, any suggestions on what I might have missed?

I added breakdownMetrics: true to the init of the Angular APM Service and our custom transactions are started by .startTransaction(name, type, {managed: true}); (just showing you that we have the managed option set to true.)

Edit: Here's an additional twist that might help. I disabled our custom transactions and I added

                breakdownMetrics: true,
                logLevel: 'trace',
                distributedTracingOrigins: ['http://localhost:4200', 'http://localhost:8080']

to the init for for the APM Service. See in this screenshot where the "click" transaction now shows a Longtask(self) between the initiating Transaction and the POST, but I was hoping to see the DNS, TCP, etc. metrics.

[vigneshshanmugam]

Hi @stitch-bb8,

Apologies for the delay.

I added breakdownMetrics: true to the init of the Angular APM Service and our custom transactions are started by .startTransaction(name, type, {managed: true}); (just showing you that we have the managed option set to true.)

Breakdown metrics for custom transactions would work when you set the managed: true based it would be based on the spans that are captured as part of transaction. so in the your above example of F752657771+0001 transaction, it would be app: x% and http: y% so it wont help you with the network level information that you are interested in.

See in this screenshot where the "click" transaction now shows a Longtask(self) between the initiating Transaction and the POST, but I was hoping to see the DNS, TCP, etc. metrics.

For transactions other than page-load, we cannot reliably calculate the DNS, TCP. etc metrics as click would have not triggered a network request and thats why we rely on calculating the breakdown based on span types as i stated above. But you have brought a good point on understanding the Network based metrics for the span and I have created an enhancement request to address it support capturing breakdown for individual spans through config · Issue #823 · elastic/apm-agent-rum-js · GitHub.

For the time being, If you are interested in understanding the non annotated times would be through UserTiming API and the agent would capture them and create spans appropriately. The same would apply to the Longtask as well as the browser is busy doing some work which would block the main thread. Check this doc which has information on interrupting long tasks.

Let us if it helps.

Thanks,
Vignesh