Invalid User Agent String

[anushshukla]

We have observed that the user agent string (user_agent.original field name) in kibana logs is showing as

fp/f99c5a

for all or any web view browsers instead of the actual user agent string.

To verify and confirm it, we have started sending the user agent string in tags of the APM transactional RUM events at client side to verify the same which we did (please refer the below screenshot).

Please help us in resolving this issue which we are facing!

[tylersmalley]

Can you confirm this data is coming from the an APM agent, if so, which one?

[anushshukla]

@tylersmalley, Yes it is coming from APM agent after we added the below configuration

`apm-server.register.ingest.pipeline.enabled: true
output.elasticsearch.pipelines:

• pipeline: "apm_user_agent"`

to apm-server.yml.

[tylersmalley]

I am going to move this to the APM group, someone there should be able to provide some more help

[simitt (Silvia Mitter)]

This indeed looks suspicious. There are currently two ways how the APM Server enriches data with the UserAgent information: It either parses the information directly from the User-Agent header from the agent request, or in case the ndjson body sent by the agent includes a User-Agent in the context.request.headers, this information is set as user_agent.original.

You mention you added the UserAgent as label. Where did you parse the information from, is it sent as headers with the agent request?

[anushshukla]

@simitt thank you for looking into this. With reference to the screenshot above in the question, the labels.userAgent value is being sent from the browser's user agent from the client side by adding it in the tags during the APM RUM transaction events. Let me know if more information is required from my side.

[simitt (Silvia Mitter)]

The APM Server doesn't process this information from labels, but either from the headers that are sent by the agent, or from whatever is sent within context.request.headers under user-agent. You should be able to find the headers in your APM Server logs. Can you confirm that the user agent information there is the same as the one you are sending in the labels?

Which versions of Elasticsearch, APM Server and the RUM agent are you running?

[anushshukla]

With references to below screenshot, the APM server responds with fp/f99c5a in it's header of server key name while in the request header of User-Agent the value is correct. I am just sending the actual user agent string (same as User-Agent request header) in tags for other purposes because due to APM server or Kibana logs, the user_agent.original field name is incorrect.

Please note that we are using https://unpkg.com/@elastic/apm-rum@4.0.1/dist/bundles/elastic-apm-rum.umd.min.js script inclusion in out frontend application which is built on React JavaScript framework.

[simitt (Silvia Mitter)]

Unfortunately I cannot reproduce this behavior locally, sending requests with some of the user-agent info showed in your examples, results in properly set user_agent information in Elasticsearch.

Do you have anything additionally set up, any proxies in between or any non-default configurations that could give a hint? And can you share which versions of the agent, APM Server and Elasticsearch you are using?

[anushshukla]

No proxy setup.

v7.1.1 is the deployment version.

The only configuration change is at in User Settings Overrides (apm-server.yml) of APM server which is

apm-server.register.ingest.pipeline.enabled: true
output.elasticsearch.pipelines:
 - pipeline: "apm_user_agent"

[simitt (Silvia Mitter)]

The issue appears when the user-agent string contains [] chars. I have forwarded the issue internally and will keep you updated.

[anushshukla]

Thank you, @simitt for your continuous support.