How to integrate apm trace id with java logs

[suresh123 (suresh)]

`Hi Team,

How to integrate apm trace id or span id with actual java logs to trace the complete java log.`

[riferrei (Ricardo Ferreira)]

Suppose the logs from the Java application are being sent to Elasticsearch already (via Filebeat or the Elastic agent). In that case, you can enable log correlation in the instrumented JVM using the parameter below:

-Delastic.apm.enable_log_correlation=true

More information here.

— @riferrei

[suresh123 (suresh)]

`Hi Racardo,

I have added -Delastic.apm.enable_log_correlation=true but getting error, please look into below

Before adding

java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1

After adding
java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1

Error
[main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
Error: Could not find or load main class jar
Caused by: java.lang.ClassNotFoundException: jar
[elastic-apm-circuit-breaker] INFO co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Stopping the Circuit Breaker thread `

[nugusbayevkk (nugusbayevk)]

@suresh123

java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1

Hi @suresh123, you make mistake in your command - before jar you need to add -

java -Xms512M -Xmx2G -javaagent:/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true -jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1

[suresh123 (suresh)]

` I added below, but I think it is not correct, can you please provide correct syntax.

java -Xms512M -Xmx2G -javaagent:-Delastic.apm.enable_log_correlation=true /services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true -jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1

Error
Error opening zip file or JAR manifest missing : -Delastic.apm.enable_log_correlation
Error occurred during initialization of VM
agent library failed to init: instrument`

[suresh123 (suresh)]

Now it is working -Delastic.apm.enable_log_correlation,
let me check the trace id is pointing to actual logs.

[suresh123 (suresh)]

Can you please guide me how to verify whether apm is integrated with actual java logs. Where can I search with trace id/span id to get complete trace log in application log-index

[riferrei (Ricardo Ferreira)]

You should be able to jump from a trace to the respective log if it is working correctly. Please note that the flag enabled in the JVM only correlates the traces with the logs, but the logs themselves need to be sent to Elasticsearch separately.

— @riferrei

[suresh123 (suresh)]

`Hi Ricardo,

We have two indexes, one is for apm-* and another is for dev1-applogs-be-*
In docker, entry point script I added like below which deployed in Kubernetes(pod is the running fine successfully)

java -Xms512M -Xmx2G -javaagent:-/services/apm/elastic-apm-agent.jar -Delastic.apm.enable_log_correlation=true -jar /services/service1/service1*.jar --server.port=<port_num> --spring.application.name=service1

index1->apm-*
These logs contain metrics such as trace id, span id and other fields.

index2-> dev1-applogs-be-*

These java application logs come from all the pods-containers across the nodes by adding log path in filebeat.yml (filebeat dameonset)

How to check complete trace log by using trace id/ span id in dev1-applogs-be-*

Please let me know if any details required`

[riferrei (Ricardo Ferreira)]

You should be able to see the logs from the trace as shown below:

But that requires the logs to be compliant with the ECS format. Make sure the logs are generated using the Java ECS Logging framework, and you will be good to go. Another way without changing your code is creating a ingest pipeline in Elasticsearch and augmenting the log data via a pipeline, which you can plug in your Filebeat configuration. But preferably, aim to generate the ECS format straight from your app.

— @riferrei

[stephenb (Stephen Brown)]

@riferrei

Make sure the logs are generated using the Java ECS Logging framework,

This is critical ^^^^

Also you can validate by just looking at the logs and you should see something like this in the logs those 2 fields get injected by the enable_log_correlation setting

trace.id : 048e8431ca04a61d0eed84e98321e122

transaction.id : 1ff61169634559c9