Distributed tracing for java application on windows system

Elastic Observability APM
, ,
1

Kibana version: 8.1.1

Elasticsearch version: 8.1.1

APM Server version:8.1.1

APM Agent language and version: Java 17 open jdk

Setup is done on Windows system and java agent of version 1.30 is used as java agent, for the application hosted on tomcat.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Hi Team,
I had setup Kelasticsearch, kibana, java agent and apmserver on my windows system, I am getting transaction details of my webapplication till controller and its endpoint i.e the method getting called on the controller. But, i am seeking to fetch the full stacktrace of the transaction like what classes are invoked, what all methods got invoked through that controller also, abstract classes and interfaces got extended and implemented respectively.

I gone through the discussion and found out that slow tracing/distributed tracing is one one option but it doesn't support windows system.

Is there any way to fetch the whole stacktrace of all transaction. Also it is observed that the stacktrace is present only for db querries taking execution time more than 5ms and if any error occurs in the transaction.

I wanted the stacktrace of all the transactions.

Thanks

2

That's not directly supported. I can think of two ways for you to achieve what you want with some work

  1. Move to running the Java application being monitored onto a Linux system (you could use a dockerized linux container running on Windows) and then use Profiling configuration options | APM Java Agent Reference [1.x] | Elastic with the min duration tuned for your application to capture all the transaction
  2. Clone the GitHub - elastic/apm-agent-java: Elastic APM Java Agent tree, edit some method that is called at the end of Transactions, eg Transaction.afterEnd() to generate a stack trace (new Exception().fillInStackTrace() then dump the trace to a file with transaction names), then build your own agent jar and run that.

The former has the advantage that it works well with the existing Kibana UI and you'll see the stack traces in the traces in the UI. The latter gives you fine control over what you want.

Hope that helps

3

Hi JAck,

I tried what you had suggested in 2nd option :
" Clone the GitHub - elastic/apm-agent-java: Elastic APM Java Agent tree, edit some method that is called at the end of Transactions, eg Transaction.afterEnd() to generate a stack trace (new Exception().fillInStackTrace() then dump the trace to a file with transaction names), then build your own agent jar and run that"

The response i got is as follows :
java.lang.Throwable
at co.elastic.apm.agent.impl.transaction.Transaction.afterEnd(Transaction.java:283)
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:496)
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:475)
at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:169)
at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:242)
at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)

It is not showing the traces of my classes or methods called in that HTTPrequest or transaction.

I tried :

  1. Thread,currentThread.getStackTrace() in afterEnd() method of transaction.java
  2. new Exception().fillInStackTrace() ; in afterEnd() method of transaction.java

Can you please suggest, how can i get it done?

4

Originally you wrote:

I wanted the stacktrace of all the transactions.

But seems you are interested in stack traces for spans. See details about the differences in our data model.

Another idea: use one of our manual tracing APIs. With that, you won't need to build an agent, but you will need to manipulate your application code, where you see fit.
For example, if you annotate some key methods in your application with our proprietary annotation API, you would get a span for the annotated method.

As you noted, the agent would collect and report a stack trace for each span that crosses a threshold, which is 5ms by default, but this is conigurable through span_stack_trace_min_duration.

I hope this helps.

5

Thanks for your input i understood the difference between transaction and spans.

Following are my requirement :

  1. I can see the stacktrace of db related spans and transactions with exception in it only.
  2. I wanted to see the stacktrace of each and every span as well as transaction.
  3. i wanted to build a flow using Elastic apm for my application for every render or transaction happens even if it is calling a method from its controller.
  4. but all i can see is stacktraces of transaction in case of exception and spans with dB related activities.

I am working on windows system with java version 17.

For now what i had done is :
In transaction.java -> beforeEnd() :
fetched activethread group and fetched the classes in following manner :

Map<Thread,StackTraceElement> activeThreadStacktrace
= Thread.getAllStackTraces();

this gave me all the stacktrace of active thread. I fear i might not get the stacktrace of thread which got end before i calling this service.

I am not sure how reliable it is. Please suggest me any improvements if any or any other way or i am heading in the correct direction.

I can not go for mannual instrumentation. Can i use instrumentation.getAllLoadedClasses() any how in elk like it can be used in glowroot?

6

Did you try setting span_stack_trace_min_duration to 0ms, as proposed in the documentation?

7

yes, i tried setting the property. This helped me in getting the stacktrace for the db spans which are not getting traced for the spans having duration less than 5 ms. but this didnot helped me in getting the stacktrace for transactions and sole java rendering.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.