Hi guys,
I noticed that body saved inside http.request.body.original field has been trunkated. Are there some options to manage this field size?
Thanks,
Andrea
Not yet, but we have discussed adding a configuration option for that. Currently it's capped at 10k chars. Could you elaborate a bit on your use case?
2 Likes
I noticed that is set to 10k and it's not due to Elasticsearch (I thought about ignore_above field property). I set apm to stdout and the original message was already trunkated.
What you mean on elaborate? Unfortunately payloads are related to some "out of the box" products and refer to order management process, so they are full of data. Is there a way to increase the size? Or do you have any other suggestion? Maybe a way to trunk white spaces indentation?
Thank you Felix
I asked for some info. The order management product original payload size is almost 800k per item. Consider that this product can managet max 10 items per product. So the payload will be 800k X 10.
Note that depending on the throughput, this can add significant overhead to record, transmit and store such large request bodies. That's why our agents truncate after 10 000 characters. Is that not a concern in your scenario?
If you can/want to talk about it I'd be interested in: Why do you want to store the request bodies? Are you doing some kind of processing/analytics on the bodies? What are you trying to achieve with storing them? Could you use the public API to add only specific labels, like the product IDs to the transactions?
Blockquote Note that depending on the throughput, this can add significant overhead to record, transmit and store such large request bodies. That's why our agents truncate after 10 000 characters. Is that not a concern in your scenario?
Yes it is. Of course I considered all those facts 
I would like to know if it's possible, anyway. Right now we are saving payload requests on an Oracle DB.
Brief explaination
Why do you want to store the request bodies -> we are developing a Fullfillment Order Management (FOM) platform which interacts with different softwares. That's why it's better to store request bodies.
Use case -> log management
Can you please explain better this part? Thanks!
What are you doing with the stored bodies? Are you interested in only specific attributes like the number of items in the cart? If so, you could use the public API to programmatically add custom labels: Public API | APM Java Agent Reference [1.x] | Elastic to the transaction.
Adding this is a bit of a dangerous feature as it's easy to shoot yourself in the foot and might even be used by malicious users to overwhelm your system. That's why I'm a bit hesitant on this. But if it's an opt-in feature and there's an appropriate warning in the documentation, I guess we can add that. Would you be interested in doing a PR for this?
I would like the ability to store larger request bodies too. We have webservice where the typical request is about 1MB and which is called approx. 300 times a day. In case the call fails we would like to store the request so we can re-execute the call. We would not have to store them as long as the rest of the APM data - 3 days should be fine for us.
As APM already creates multiple indexes(for spans, transactions,...) - maybe it is possible that the requests are also stored separately so we could delete those indexes sooner than the rest.
Another thought on this: Currently, the apm agent stores the request(or not) only based on the configuration settings. Maybe it is possible to provide either a method so the application can manually set the request data so the application has control whether the request should be stored or not. This would greatly reduce the fear of malicious usage as the application could validate the request. I do not mean that the application would validate the request for logical correctness as we want to find the errors of requests too of course - I mean that the application could stop the agent to store requests of unauthorized users or requests larger than a plausibility limit or such things.
1 Like
Stored bodies are useful if they need to be checked in order to verify/control their consistency through the FOM (consider that more than 8 different products are used and integrated through the business process). We are definetely interested in preserving them.
Right now I'm trying to create custom transaction (for other purposes) but I'm facing some issues. I will open another thread to keep things separated.
Thanks
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.