I am using Elasticsearch version 7.17.1 and APM agent version 1.28.4
When I use Spring Boot 2.6.3 with APM agent, the request body can be shown on the metadata tab of the error page.
However, when I updated the Spring Boot to 2.6.4 or above, the error page cannot show the request body. I only can find to request body on the transaction page.
I have tried to set the capture_body to transactions, errors and all, but the error page also cannot show the request body.
Can I fix it through updating the version of Elasticsearch or APM agent? or any another solutions for this problem?
Hi, could you give an example screen capture of the data showing? And ideally a simple set of instructions to reproduce what you are seeing in 2.6.3 that no longer shows in 2.6.4 (ideally a simple app that shows this)? Thanks
Hi Jack. Thanks for your reply. The following are the steps to reproduce the issue.
Test with spring-boot 2.6.4+
- Download the demo project at GitHub - davidlee950628/elk-spring-boot-demo
- Following the README.md to send the requests to ELK
- You should be able to see these two errors in the APM page
- Click the error "test string request body exception", then view the tab "Metadata"
- The request body of error "test string request body exception" can be displayed successfully
- Back to the APM page, then click the error "test object request body exception" and view the tab "Metadata"
- The request body of error "test object request body exception" cannot be displayed
Test with spring-boot 2.6.3
- Change the version of spring boot in the file build.gradle
- Build the jar again
- Following the README.md to send the requests to ELK
- Now check the "Metadata" tab of the error "test object request body exception". The request body can be displayed
Now, we see the difference of these two APIs:
For the API /test-string-request-body, I use the annotation @RequestBody with String, so it can work in both version of Spring Boot
For the API /test-object-request-body, I use the annotation @RequestBody with self-defined class, so it cannot work in Spring Boot 2.6.4+
My conclusion is that the annotation @RequestBody cannot work with APM agent appropriately after Spring Boot 2.6.4+
Thanks (sorry for the slow responses, it's holiday season but quite a lot of things cropping up). I hope to be able to try this out, but in the meantime did you check the agent log in DEBUG mode? It might show an error against that request
Hi Jack. I have tested and no error logs from the agent
Hi @David_LEE1 ,
Good news, I've managed to reproduce the issue locally, thus it should be something we can fix quickly.
There are a few different issues here
- for the "object body" variant, the body is missing from the error document, but not from the transaction document (exactly what you reported).
- for the "string body" variant, everything is serialized in the property name
http.request.body.original.{"name": "test"} in the transaction document.
I'm using a 8.x apm-server and latest agent, though. Can you tell us if the second issue also happens in your case ?
Actually this second issue is very probably not a valid one as it is triggered when the Content-Type header is missing, it seems that the default is to assume application/x-www-form-urlencoded and because the agent does not parse it as JSON (but copies it as-is), it creates this weird attribute name.
Hi,
First, I forgot to thank you for providing such an easy way to reproduce your issue, taking the time to do it saves a lot on our end, thus thanks a lot for doing that.
Regarding the content-encoding, the application/x-www-form-urlencoded comes from curl by default when no explicit value of Content-Type is set, thus it is not related to the agent nor the application (thanks @felixbarny for checking that).
For the error capture, the request body is only missing from the Error document, but is properly captured for the linked Transaction. In UI, navigating from Error to the Transaction should allow you to find the request body that you expect on the corresponding Transaction document.
On a more technical note, this is only due to how error capture works in this particular case:
- Errors (exceptions) are captured and sent immediately by the agent
- When an Error happens within a transaction (in your case), parts of the in-progress
Transaction is copied to create and send the Error to APM-server, thus relying on partial or transient data.
- When using a "string" argument, the request body is captured early and as-is from an immutable
String, so request body captured on the transaction and the error are consistent.
- When using an "object" argument, the request body is captured by wrapping the
InputStream from the request, which means the captured value is only considered complete when the InputStream#close method is invoked, which might happen AFTER the exception is thrown.
- Changing this behavior could lead to tricky corner cases and unexpected consequences: buffering overhead and extra complexity that might not be worth the effort.
In other words, relying on Transaction captured body should be the most reliable approach, request body that might be shown in Error is only available as a best effort and might not be always available, at least for the Java agent.
In UI, in order to navigate from the Error to the related Transaction, you have to use the link at the bottom right of this screenshot:
Hi,
Thank you for your reply. I can see the request body in the transaction document. However, my company doesn't capture all transactions, so sometimes I cannot find transaction documents. Is it normal to capture all the transactions for most of the companies?
Hi,
By default the agent captures 100% of the transactions, but it is common to change this default by using the transaction_sample_rate configuration.
However, capturing the request body for all transactions combined with 100% sample rate will definitely store a lot of data in ES which is likely to create performance issues on the APM/ES side.
I don't have sound data on this, but my gut feeling is that the body capture is disabled for most users, a few subset of them enable it for error transactions to help investigate them, and capturing for all transactions is only used rarely to provide a kind of "debugging".
Thus depending on what you are trying to achieve, there are different possible strategies (non-exhaustive list)
- minimize apm data usage: disable request body capture + use < 100% sample rate
- moderate apm data usage: disable request body capture + 100% sample rate
- debug only transactions with errors: enable body capture on errors + use 100% sample rate
- debug everything (on a limited scope, for example outside of production): enable request body capture on all transactions + use 100% sample rate
Here the "data usage" and actual sample rate will depend on your application, the load it receives and how much resources are available on APM Server/Elasticsearch side, thus I would suggest to experiment and measure in order to make sure it fits your usage/setup.
Hi,
Thank you for your detailed answer. It helps me a lot.
