Kibana groups transactions under generic GET and POST paths

[gmacini (Guido Macini)]

Hi,

We are trying your technology to monitoring a cloud service deployed on Kubernetes and developed using Vert.x.

We enabled the experimental instrumentation (following the instructions of this link) because the default configuration of the APM agent wasn't working properly.

With the experimental instrumentation, the reports created in Kibana only show the generic rest paths which are different from the paths of reports created using the default instrumentation.

As you can see on the screenshots attached, we obtain:

• one overview for all the GET transactions;

• one overview for all the POST transactions;

Thanks

[Eyal_Koren]

@gmacini welcome to our forum and thanks for reporting!

@gmacini

the default configuration of the APM agent wasn't working properly

Please provide some more info - what wasn't working properly? Did you not get transactions you expected? Did you get transactions with improper name/duration/other?

@gmacini

With the experimental instrumentation, the reports created in Kibana only show the generic rest paths which are different from the paths of reports created using the default instrumentation.

Please provide 2-3 examples of paths you expected to get. What paths are you using? Are you relying heavily on : path parameters or any other Vert.x specific features?

Finally and most importantly, please provide the following information:

• what Vert.x version are you using?
• based on your screenshot, your system contains some integration with Servlets - please describe what exactly you are using
• please configure log_file to get a separate agent, set log_level=debug and share a log that starts with the application startup and continues until some requests are being captured. You can share the log through gist.

[gmacini (Guido Macini)]

Hi Eyal and thanks for your quick answer!

The default configuration, without the experimental_instrumentations doesn't monitor the rest calls as in the following screenshot

When I enable the experimental_instrumentations, setting to true ELASTIC_APM_ENABLE_EXPERIMENTAL_INSTRUMENTATIONS, the system groups all transactions under generic GET and POST paths, as in the following screenshot

But, what I want is the following behavior, obtained with an old version of my app before introducing of reasteasy-reactive and, consequently, vertx

I attach 2 more screenshot to show the monitoring of an endpoint to persists a folder (call: POST http://localhost:8080/rest/drive/elements?operation=folder):

1. wrong behavior of current app with group calls

2. expected behavior of old app with split calls

All the calls have the same problem, as e.g.

• GET http://localhost:8080/rest/drive/sections/starred
• GET http://localhost:8080/rest/drive/sections/myfiles

We are not relying heavily on path parameters or any vertx feature.

The vertx version is 4.3.4

We use a servlet to perform logout removing cookies and redirect to login page and some http filters to redirect users to certain pages based on specific conditions

Here the log file with debug level (I perform a POST to create a folder)

[Eyal_Koren]

Thanks for sharing this useful info.
From the log I can see that basically all Vert.x instrumentations are applied and I can't see any faults.

Here are some thoughts:

• If I am not mistaken, RESTEasy Reactive is a JAX-RS implementation, which means that your code uses JAX-RS annotations, which are then picked up and mapped into Vert.x routes (I am not familiar with that, only guessing here). If that's the case, it could be that our JAX-RS instrumentation "competes" with our Vert.x instrumentation, so here are two proposals to try out:
  1. Rely only on the JAX-RS instrumentation: remove the ELASTIC_APM_ENABLE_EXPERIMENTAL_INSTRUMENTATIONS setting and verify that your application_packages are defined to a root package that includes ALL classes that contain the JAX-RS annotations
  2. Rely only on the Vert.x instrumentation by adding the following config: disable_instrumentations=jax-rs
• Try setting use_path_as_transaction_name=true, to tell all instrumentations that you want to rely on the request path for transaction naming. You say that you don't rely on path parameters, but notice that if you do even by one route, you should configure transaction_name_groups as well, otherwise your transaction group cardinality will be too high

I will be looking forward to hear your feedback.

If I am way off, please provide an example of how your endpoint code looks like. Doesn't have to be an exact class you use, could (actually - better) be a minimal example.
If you provide a minimal but fully functional project (e.g. through GitHub) that we can use to easily reproduce what you are experiencing, that would give the highest chances of us being able to assist.

[gmacini (Guido Macini)]

I tried your suggestions but it still doesn't seem to work.
I created a github project at this link, and in the README.md file I added some information about my environment

[Eyal_Koren]

Thanks for providing the test project!
So I assume it is a general matter of support for RESTEasy-reactive.
It may take us some time, but we will look into that once we have a chance.
Worth saying in advance that if this is more than just a fix/enhancement to existing instrumentations and requires a new plugin, we may not be able to prioritize it, in which case we will propose an alternative.

[gmacini (Guido Macini)]

Thank you!
We await your reply as we would like to adopt your solutions into our products!

[Eyal_Koren]

Understood. We will be happy to assist as much as possible.
We didn't have the chance to look deeper into this so far.

[Jonas_Kunz]

I have been able to reproduce the problem with your provided test application, thanks for your help!

I can confirm Eyal's hypothesis that our Vert.x instrumentation competes with our JAX-RS instrumentation for naming the transactions.

I created a GH issue for it, you can follow the issue to track the progress.

[gmacini (Guido Macini)]

Hi Jonas,
thanks for help!