Elastic Observability On-Demand: APM RUM agent seems to have wrong version

[jan.stap]

Hi,

Lab 3.4 instructs to install version 4.9.0 of the APM RUM agent by running:

npm install @elastic/apm-rum@4.9.0 --save

After doing this and then applying the agent configuration as instructed in the lab I do receive the agent data in Kibana, however the Service Map has no connection between the RUM agent and the Node.js agent (marked with red cross):

The Petclinic application itself works fine as seen from my browser, so data is flowing from the React to the Node.js application.

I notice that the APM integration in Kibana has version 8.2.0 and the APM agent compatibility page says that for APM integration version >= 7.0 one needs to use a JavaScript RUM agent version 5.x, not 4.x.

So... can I fix this issue by upgrading the RUM agent to the latest version? (currently 5.12.0).

Thanks!

[Andre_Murbach_Maidl]

Hi @jan.stap ,

the issue is actually related to the way the load generator works, as most of the transactions are triggered from petclinic-node. After you interact with PetClinic in your browser you can narrow the time window with the time picker (e.g. Last 15 minutes) and you will probably get the proper service map. We will work to fix this in a next training release. Thanks for pointing this out.

Best,
Andre

[jan.stap]

Hi Andre,

I do have used the Petclinic GUI quite a lot and in this way generated traffic via the React component of the app. But still no connection in the Service Map and no transactions spanning from React onwards through Node.js and on.

I did have an issue initially with the petclinic-java component as I made a backup copy /root/start-scripts/50-petclinic-server.sh-20221022 of the /root/start-scripts/50-petclinic-server.sh file, not smart, as this resulted in two copies trying to start. So I deleted the backup file and restarted petclinc-java, after which it worked fine, also the Java agent.

This caused the RUM agent to send data before the Node.js agent did. Could this have caused the "missing link"?

And what about the compatibility page saying we need a 5.x RUM agent?

Thanks.

[jan.stap]

Hi Andre,

I zoomed in on the period of my own activity on the petclinic GUI, but still get the same result. Tried couple of times.

Trying to figure out in the traces-apm.rum-* datastreams what may be wrong. At least all event.outcome field values are "unknown"; for non-rum they are mostly "success". Docs here don't go into that detail, sigh...

Cheers,
Jan

[jan.stap]

Hi Andre,

Please reset my labs, such that I can hopefully get a working setup regarding RUM.

My subscription email is: jan.stap@atos.net

Thanks!
Jan

[Andre_Murbach_Maidl]

Hi @jan.stap ,

I already replaced your lab instance.

Best,
Andre

[jan.stap]

Thanks!

[jan.stap]

Hi Andre,

I have run through all labs again up to and including the RUM agent config. I still get an isolated RUM agent in my service map. Done a lot of accesses on the Petclinic GUI, in Kibana APM selected just that timeframe and still get an isolated RUM agent. Tried other browser (Chrome instead of Firefox), tried Dynamic DNS name instead of public URL for apm_server_js param (but resolves to same IP, so indeed makes no difference).

Any suggestions are welcome. But I don't want to burn all my lab time on this topic, so I continue the labs.

Thanks,

Jan

[Andre_Murbach_Maidl]

Hi @jan.stap ,

can you please share your configuration file?

Best,
Andre

[jan.stap]

Hi Andre,

This is my config.js:

var config = {
  apm_server: process.env.ELASTIC_APM_SERVER_URL || 'http://fleet-server:8200',
  apm_server_token: process.env.ELASTIC_APM_SECRET_TOKEN || 'nonprodpwd',
  apm_server_js: process.env.ELASTIC_APM_SERVER_JS_URL || 'http://ec2-52-59-226-231.eu-central-1.compute.amazonaws.com:8200',
  apm_service_name: process.env.ELASTIC_APM_SERVICE_NAME || 'petclinic-node',
  apm_client_service_name: process.env.ELASTIC_APM_CLIENT_SERVICE_NAME || 'petclinic-react',
  apm_service_version: process.env.ELASTIC_APM_SERVICE_VERSION || '1.0.0',
  api_server: process.env.API_SERVER || 'http://petclinic-server:8000',
  api_prefix: process.env.API_PREFIX || '/petclinic/api',
  address_server: process.env.ADDRESS_SERVER || 'http://address-finder:5000',
  distributedTracingOrigins: process.env.DISTRIBUTED_TRACINGS_ORIGINS || 'http://petclinic-client:4000,http://petclinic-server:8000,http://localhost:4000,http://localhost:8080,http://localhost:8081'
}

module.exports = config;

Basically the same as in the lab solution.

Cheers,
Jan

[Andre_Murbach_Maidl]

Hi @jan.stap ,

is ec2-52-59-226-231.eu-central-1.compute.amazonaws.com your current Public DNS? The config file should be using the current Public DNS of your lab instance.

Best,
Andre

[jan.stap]

Hi Andre,

Yes, I update that every time it changes after pause/resume of my labs, after which I restart the petclinic-client container.

FYI in Kibana Discover I found that the RUM agent just sees the loadgen traffic, not my browser traffic:

• my FireFox and Chrome browsers do appear in the set of user agents:

• but when filtering on the FireFox one, I only get these APM agent names:

• disabling this filter also gives me agent.name:"js_base", which must be the RUM agent. Filtering on that agent, I only get the loadgen-related user agent, so no FireFox or Chrome:

So in all, the RUM agent just sees the loadgen traffic, whereas the other three agents see loadgen, FireFox and Chrome traffic. Could this be some browser security filter blocking JavaScript injection? (not sure if that happens at all here though).

Thanks,

Jan

[Andre_Murbach_Maidl]

Hi @jan.stap ,

I tested on my end and it worked. Are you accessing PetClinic using the following URL pattern?

http://<public-DNS>:4000

If you use the internal Strigo URL it will not be able to communicate with the RUM agent.

You can also use the developer tools of your browser to check if your APM agent is being created correctly. You should see something like this:

If you see any error that's an indication that something is not right in the configuration.

If you still have trouble you can send me a private message with your Public DNS so I can take a look in your environment.

Best,
Andre

[jan.stap]

Hi Alex,

Got it! Should have remembered myself to check devtools. The agent was blocked due to a "mixed content" web page: the app itself is served over HTTPS, but the agent connects over HTTP. I got this:

and:

The "[Learn More]" in the top screenshot leads to here. Since FireFox 23 mixed content is blocked, the page says. I see the protection can be manually switched off for the session duration:

After doing so, I indeed get the correct service map:

Thanks a lot for your help! I guess this info is something to include in the labs, or even better the agent to be put on HTTPS.

Best regards,

Jan

[jan.stap]

FYI in Chrome 106.0.5249.120:

Cheers, Jan

[Andre_Murbach_Maidl]

Hi @jan.stap

thanks very much for your feedback. We will indeed add this note to lab instructions about using HTTP over HTTPS.

Best,
Andre