Some transaction names are incorrect/falsely reported

[schang]

Kibana version:
7.15.2

Elasticsearch version:
7.15.2

APM Server version:
7.15.2

APM Agent language and version:
Java 1.28.4

Browser version:
Chrome 100.0.4896.75

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Download page

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?

We have elastic.apm.enable_experimental_instrumentations enabled, as all our services use the Vert.x framework

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Most transaction names for our endpoints are reported correctly as registered in our vert.x routes, e.g.
GET /v1/customers/:customerId
GET /health

However, every now and then, a transaction name will be reported with the valid transactions (see screenshot above), with transactions such as:

GET /v1
GET /v1/
GET null

This doesn't happen all the time, but on a semi-frequent basis.

[Eyal_Koren]

Welcome to our forum and thanks for reporting!
Please provide the following details, so we can investigate:

• Your Vert.x version
• Are you using HTTP 1 or 2?
• Your full agent setup (installation method and configurations)
• A debug log. In order to produce such, stop the app, clear existing logs, set log_level=debug, restart and share the resulted log from startup and until such transaction is captured and reported (debug log would include this info). You can share the log through gist.

[schang]

Hi there! Thanks for the response.

• Vert.x version: 3.9.9
• Should be HTTP 1.1? We don't set it in our server configs, so it's default
• Installation is from Maven here: Maven Central Repository Search.
• Configurations are like so:

	-Delastic.apm.enable_experimental_instrumentations="true" \
	-Delastic.apm.service_name=$SERVICE_NAME \
	-Delastic.apm.environment=$ENV \
	-Delastic.apm.sanitize_field_names="password, passwd, pwd, secret, *token*, *session*, *credit*, *card*, authorization, set-cookie" \
	-Delastic.apm.service_version=$(cat build-number.file) \
	-Delastic.apm.server_urls=$ELASTIC_APM_URL \
	-Delastic.apm.secret_token=$ELASTIC_APM_TOKEN \
        -Delastic.apm.application_packages=[[packageNameHere]]

I can try to get the logs, but this seems to only happen more frequently in our production environment and not consistently, so not too sure how feasible that is. Might take a few days to get some logs from dev. Will update with logs as soon as I can.

[schang]

Hello -- here are some logs, as requested.

https://gist.github.com/schang1-godaddy/f7143ffae418d62856fac323445baec6

Let me know if you need more info! Thanks.

[Eyal_Koren]

Thanks for the info and the filtered log!

I think there are two separate issues here:

1. Requests for non-existing routes: those should have all been aggregated under the same transaction name <HTTP-METHOD> unknown route, however your log shows that this changes to <HTTP-METHOD> null during request processing. I will look into it and see why that is. In order to verify that your GET null transactions are such indeed, please click into them and look in the Transaction Details, as shown below. See if the http.response.status_code for these is 404 or other. Look also for other relevant info about these requests to understand where they come from (e.g. user_agent and host fields) and what route they were aiming to hit (through the url fields). You may want to try out the use_path_as_transaction_name config. With that enabled, these transactions should have a more informative name. On the other hand, if there are lots of requests to different non existing routes, each will have a different transaction name, causing transaction clutter.
2. Requests for non-full routes (like GET /v1): please look into these as described for the former and see what are the related HTTP requests that are causing this- origin host, status code, url.full, user_agent etc. Try to combine that with your knowledge of the application and see if you can provide more details. If you find debug log for such example as well, that may be very useful.

[Eyal_Koren]

I could only see one reason for these GET null transactions, which I addressed, but I couldn't reproduce, so I can't be certain about it.
Please try out this snapshot and let us know whether at least those are eliminated.
Since I applied this null check on a bigger Vert.x-related PR, which contains some more central changes to the Vert.x instrumentation, please make sure that everything works as expected. The changes to Vert.x 3 with HTTP 1 are quite minor, so I assume it should be good.

[schang]

Thanks so much!

I tested out the SNAPSHOT jar in dev and let it bake for a day and that does seem to fix the GET null errors. Please let me know when the SNAPSHOT changes are fully released to be used in production and the new version!

I think in the meantime, we'll set the transaction names in each handler and see if that resolves the other, pesky vague route names temporarily. We considered use_path_as_transaction_name, but since we don't have as many routes, it might be easier to manually set the transaction name in each handler.

All the GET /, POST unknown route, GET unknown route transactions (which appeared after I deployed the new jar in place of GET null and POST null) are 3xx and 4xx, so that makes sense to me. GET /v1/, GET /v1 transaction names in Elastic APM are all 2xx, however, and that might require some more digging.

I'll definitely dig to see if I can find more logs regarding the non-full routes and update this thread. Thanks!

[Eyal_Koren]

Thanks for the update!

@schang

Please let me know when the SNAPSHOT changes are fully released to be used in production and the new version!

I will soon merge the related PR and it will be included in the next release. We don't have a target date for that yet, but you can watch our repo to get notifications specifically on new releases.

As for the 2xx with the unclear route, did you look in the Transaction Details as proposed above to see if you can get hints about the source for these?