I've set up an ELK stack on Docker which I primarily use for ingestion and analysis of logs produced by a Java application. The logs are fed to ES by LogStash (file input) which works well. I have also added --javaagent JVM options on the Java application to be able to collect logs/traces/api data/etc. The agent will receive configuration information from APM Server which appears to work as well as the following is evident on logs:
Received new configuration from APM Server: {api_request_size=1024kb, capture_body=all, capture_headers=true, circuit_breaker_enabled=true, enable_log_correlation=true, log_level=trace, profiling_inferred_spans_enabled=true}
On Kibana, under APM -> Services, I only see JVM Metrics data collected. I don't see any data on Overview/Transactions/etc. Perhaps this is a basic question but do I need to make code changes to the Java application to be able to view data here? Ideally I was hoping to see detailed thread information, API call information/etc. TIA.
Note that because my application runs on Windows, aysnc-profiler is apparently not supported.
For completeness-sake, this is my local configuration of the agent which is believed to be overridden with the remote configuration data from APM Server:
Have you tried executing a few actions on your application ? For example if it's an HTTP server then executing a few queries to the application endpoints should be enough.
The agent passively captures transactions and spans, thus if there is no activity that triggers a transaction, then no trace is being created. The metrics are sampled and sent on a regular basis, so even without any activity on the application they are reported and visible in Kibana.
Last but not least, application logs alone do not allow to create transactions, but capturing them allows to correlate them with the traces (transactions & spans).
@Sylvain_Juge - thx for the note. Yes, I did invoke the API hosted on the application. So, there is certainly activity on the JVM.
As for the below remark, I believe the config options sent (seen above) by the APM server does instruct capturing the requests/etc. Or do I understand incorrectly? Please advise.
but capturing them allows to correlate them with the traces (transactions & spans)
On a different note, I see the following on log, not sure if this has to do anything with what's (not) evident:
2022-12-15 11:42:06,043 [Thread-14] TRACE co.elastic.apm.agent.bci.ElasticApmAgent - Advice co.elastic.apm.agent.slf4j.**error**.Slf4jLoggerErrorCapturingInstrumentation is loaded by ShadedClassLoader{parent=ShadedClassLoader{parent=jdk.internal.loader.ClassLoaders$PlatformClassLoader@2a266d09, customPrefix='cached-lookup-key/', manifest=java.util.jar.Manifest@19a1201d, jarUrl=file:/C:/elastic-apm-agent-1.35.0.jar}, customPrefix='agent/', manifest=java.util.jar.Manifest@19a1201d, jarUrl=file:/C:elastic-apm-agent-1.35.0.jar}
For debugging any issues with agent, most of the time DEBUG is enough, TRACE is quite verbose thus you only have to use it if really needed. Most TRACE message are very low-level implementation details.
It looks like the application uses custom code that is not on the supported technologies list. We will make use of plug-in api for instrumenting... Thank you.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.