User can't view APM services in Kibana

[qd-danh (Dan H)]

We are new to sending APM data from our services to elastic (using Serilog and .NET libraries). It seems to be working fine. I am admin and can see our sample service listed in APM, Services and then can drill down into dependencies, errors, metrics, all is good.

However, the devs on our team (not admin) get a kind of "welcome" message as if there is no APM data at all. Same instance, same data, just they are "blocked" from seeing the data in the UI.

It feels like a permissions or privilege problem, since I know the APM data is there. I'm digging through docs, privileges, and even elastic consulting doesn't know what's missing in our setup.

Any ideas what else to try?

Screenshot below from admin user - CAN see the data.
Screenshot below from dev user - CAN NOT see the data.

Thank you!

[lwintergerst (Luca Wintergerst)]

Hello Dan,
you are right, this likely has to do with some missing privileges for the "dev" user.

If you want me to help you with the troubleshooting, can you login as one of the dev users and then run the following and send me the output?

GET _security/_authenticate?filter_path=roles

We'll likely need to go back and forth a few times. Once I have the output of the above I might need other information from you.

Alternatively you could try to follow this guide here and assign the newly created role to all your dev users:

APM reader user | Kibana Guide [8.8] | Elastic

[qd-danh (Dan H)]

Thank you Luca. Replied to you with direct message.

I will update this thread with any progress we make, to help others that might have this same issue.

[qd-danh (Dan H)]

Update here after working with Luca...

There are two ways to address this issue.

• In the role, add the read and view_index_metadata privileges to the following indices:

  • traces-apm*
  • apm-*
  • logs-apm*
  • metrics-apm*
  • do NOT use the suggested drop down item that looks like all of these together. They all get added as a single string, instead of individual array items, and doesn't work. Add each index pattern individually to the list (Screenshot below)
  • documentation here (but doesn't talk about the suggested drop down item not working)

• add the built-in viewer role to the role or user. This is the new way, replacing apm_user as suggested in older documentation.

  • caution: this gives the user / role read permissions to every index in your cluster that does not start with a . this may or may not be what you want

Hope that helps.
(@lwintergerst feel free to chime in with anything else - thank you for the help)