`Hi Team,
How to integrate apm trace id or span id with actual java logs to trace the complete java log.`
Suppose the logs from the Java application are being sent to Elasticsearch already (via Filebeat or the Elastic agent). In that case, you can enable log correlation in the instrumented JVM using the parameter below:
-Delastic.apm.enable_log_correlation=true
More information here.
`Hi Racardo,
I have added -Delastic.apm.enable_log_correlation=true but getting error, please look into below
Before adding
java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1
After adding
java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1
Error
[main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
Error: Could not find or load main class jar
Caused by: java.lang.ClassNotFoundException: jar
[elastic-apm-circuit-breaker] INFO co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Stopping the Circuit Breaker thread `
Hi @suresh123, you make mistake in your command - before jar you need to add -
java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true -jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1
` I added below, but I think it is not correct, can you please provide correct syntax.
java -Xms512M -Xmx2G -javaagent:-Delastic.apm.enable_log_correlation=true /services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true -jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1
Error
Error opening zip file or JAR manifest missing : -Delastic.apm.enable_log_correlation
Error occurred during initialization of VM
agent library failed to init: instrument`
Now it is working -Delastic.apm.enable_log_correlation,
let me check the trace id is pointing to actual logs.
Can you please guide me how to verify whether apm is integrated with actual java logs. Where can I search with trace id/span id to get complete trace log in application log-index
You should be able to jump from a trace to the respective log if it is working correctly. Please note that the flag enabled in the JVM only correlates the traces with the logs, but the logs themselves need to be sent to Elasticsearch separately.
`Hi Ricardo,
We have two indexes, one is for apm-* and another is for dev1-applogs-be-*
In docker, entry point script I added like below which deployed in Kubernetes(pod is the running fine successfully)
java -Xms512M -Xmx2G -javaagent:-/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true -jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1
index1->apm-*
These logs contain metrics such as trace id, span id and other fields.
index2-> dev1-applogs-be-*
These java application logs come from all the pods-containers across the nodes by adding log path in filebeat.yml (filebeat dameonset)
How to check complete trace log by using trace id/ span id in dev1-applogs-be-*
Please let me know if any details required`
You should be able to see the logs from the trace as shown below:
But that requires the logs to be compliant with the ECS format. Make sure the logs are generated using the Java ECS Logging framework, and you will be good to go. Another way without changing your code is creating a ingest pipeline in Elasticsearch and augmenting the log data via a pipeline, which you can plug in your Filebeat configuration. But preferably, aim to generate the ECS format straight from your app.
1 Like
This is critical ^^^^
Also you can validate by just looking at the logs and you should see something like this in the logs those 2 fields get injected by the enable_log_correlation setting
trace.id : 048e8431ca04a61d0eed84e98321e122
transaction.id : 1ff61169634559c9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.