Welcome to our forum and thanks for reporting!
Please provide the following details, so we can investigate:
Your Vert.x version
Are you using HTTP 1 or 2?
Your full agent setup (installation method and configurations)
A debug log. In order to produce such, stop the app, clear existing logs, set log_level=debug, restart and share the resulted log from startup and until such transaction is captured and reported (debug log would include this info). You can share the log through gist.
I can try to get the logs, but this seems to only happen more frequently in our production environment and not consistently, so not too sure how feasible that is. Might take a few days to get some logs from dev. Will update with logs as soon as I can.
Requests for non-existing routes: those should have all been aggregated under the same transaction name <HTTP-METHOD> unknown route, however your log shows that this changes to <HTTP-METHOD> null during request processing. I will look into it and see why that is. In order to verify that your GET null transactions are such indeed, please click into them and look in the Transaction Details, as shown below. See if the http.response.status_code for these is 404 or other. Look also for other relevant info about these requests to understand where they come from (e.g. user_agent and host fields) and what route they were aiming to hit (through the url fields). You may want to try out the use_path_as_transaction_name config. With that enabled, these transactions should have a more informative name. On the other hand, if there are lots of requests to different non existing routes, each will have a different transaction name, causing transaction clutter.
Requests for non-full routes (like GET /v1): please look into these as described for the former and see what are the related HTTP requests that are causing this- origin host, status code, url.full, user_agent etc. Try to combine that with your knowledge of the application and see if you can provide more details. If you find debug log for such example as well, that may be very useful.
I could only see one reason for these GET null transactions, which I addressed, but I couldn't reproduce, so I can't be certain about it.
Please try out this snapshot and let us know whether at least those are eliminated.
Since I applied this null check on a bigger Vert.x-related PR, which contains some more central changes to the Vert.x instrumentation, please make sure that everything works as expected. The changes to Vert.x 3 with HTTP 1 are quite minor, so I assume it should be good.
I tested out the SNAPSHOT jar in dev and let it bake for a day and that does seem to fix the GET null errors. Please let me know when the SNAPSHOT changes are fully released to be used in production and the new version!
I think in the meantime, we'll set the transaction names in each handler and see if that resolves the other, pesky vague route names temporarily. We considered use_path_as_transaction_name, but since we don't have as many routes, it might be easier to manually set the transaction name in each handler.
All the GET /, POST unknown route, GET unknown route transactions (which appeared after I deployed the new jar in place of GET null and POST null) are 3xx and 4xx, so that makes sense to me. GET /v1/, GET /v1 transaction names in Elastic APM are all 2xx, however, and that might require some more digging.
I'll definitely dig to see if I can find more logs regarding the non-full routes and update this thread. Thanks!