Custom stacktrace hampering the json format for long text size

Elastic Observability APM
1

HI Team,
I had added a custom change in elastic apm agent's source code where in : \apm-agent-core\src\main\java\co\elastic\apm\agent\impl\transaction\transaction.java -> beforeEnd()

to get all the active threads in jvm to get the stacktrace for java apm-agent in windows Operating System.

This code change is giving me stacktrace of all the java files , methods, line numbers.

Can you please provide your thoughts on this, if it is effective way to get the stacktrace of java files in windows OS.

Also, can you please guide me on following issue :

  1. the stacktrace is getting recorded but the json of transaction is as follows (copied the relative json only.):

"transaction" : {
"result" : "HTTP 2xx",
"duration" : {
"us" : 522004
},
"custom" : {
"stacktrace" : """[{"fileName":"CmnUserServiceImpl.java","methodName"…"""
},
"name" : "CmnAppAgentDtlController#fetchAllAppAgentsDtls",
"span_count" : {
"dropped" : 4,
"started" : 11
},
"id" : "ec82e44066dadd75",
"type" : "request",
"sampled" : true
}

  • In the methodName, the rest of the json is getting truncated and the json is not completely loading hence tampering the json format.
2

Hi @yahire ,

Could you help me understand what you are trying to achieve here exactly ?
Do you want to capture the stack trace of all running threads ?

This part of the agent is not OS specific, thus there is nothing particular with Windows here.

There is a limit on the JSON payload size when sending this to the APM server, so by setting a custom field to contain stack traces of multiple threads you are hitting the limit.

3

If I am not wrong, Slow tracing is not supported with elastic apm-agent on windows operating system.

I tried tracing the application but slowtracing was not working, I was not getting the interfaces and abstract classes traced in the stacktrace.

So, as a work around i tried fetching all the active threads and there i found the interfaces and abstract classes from active thread's stacktraces.

Regarding, JSON Payload size, is there any workaround? what is the maximum size for the JSON payload?

4

Hi,

What you seem to refer as "slow tracing", is actually "sampling profiler", which is an alternate way to capture slow parts of the code to complement tracing which is provided through automatic instrumentation.

As explained briefly here, the sampling profiler should only be relevant to create spans by sampling the threads and correlate them to an active transaction. Unfortunately because it relies on async-profiler this feature is not available on Windows.

What you seem to be trying to do here is to capture the thread state of all the active threads within a single transaction, probably with only a single sample, which means that you'll get both too much data to serialize (the spans created by the sampling profiler are "flattened"), and you will have only a single sample whereas the sampling profiler captures them with a rather high frequency (I don't recall the exact default value here).

If you really want to implement such a feature on Windows by iterating over all the other threads, you'll have to do the following:

  • create an agent plugin, probably more convenient if it's an external one
  • sample the thread states at a regular interval
  • correlate the stack traces to the active transactions and spans
  • re-construct inferred spans from the samples like the sampling profiler does

So, while it's definitely doable, this could be quite an extensive task, but it could also be worth it if you really need this and your production environment is on Windows.
If on the other end it's only for local development or testing, then I would suggest trying to run your application under linux if you can, using Docker or Windows Subsystem for Linux (WSL) would be the most convenient options.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.