Wildfly 10 trace.id and transaction.id

Elastic Observability APM
1

APM Agent language and version: Java Agent 1.16.0

Hi,

we are currently setting distributed tracing between our µservices and our legacy application.

Our µservices are based on spring boot. So no real problem excepted that WebFlux is not currently supported.

But the legacy application is an ear application deployed on a WIldFly 10.

So the problem is that our logs don't contain the trace.id or transaction.id

We can see all the transactions in kibana from REST to Database transactions.

But it seems the trace.id and transaction.id are not put in our MDC.

We followed the steps to configure the WF in domain mode.

We added the MDC part to the log4J pattern layout as for our µservices : [trace.id:%X{trace.id}, Transaction.id:%X{transaction.id}] ........

But both logging information remain empty:

Here is an example of our log output:

[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30)  26-05-2020 12:41:14.014 INFO [H|buyway-internet|L171212|d31ab668-15b3-4fdf-a39b-8bdabd958209|0000|-] (ContainerResponseFilterWrapper.java:45) - * Client out-bound request
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > GET http://localhost:8080/bwwebservices/products/?retailerNumber=1989151&productCode=PAT&creditAmount=2500.00&mode=OPENING&filterRateRanges=true
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > Accept: application/json
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > Accept-Language: fr
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > Authorization: Basic Yn... <truncated>
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > Connection: keep-alive
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > Content-Type: application/json
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > elastic-apm-traceparent: 00-537fc812faa5fd84567ae0bc3af80953-131edc8b70f4f838-01
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > Host: localhost:8080
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > traceparent: 00-537fc812faa5fd84567ae0bc3af80953-131edc8b70f4f838-01
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30) > User-Agent: Java/1.8.0_144
[Server:phenix-local] [trace.id:, transaction.id:] 12:41:14,013 INFO  [stdout] (default task-30)

I can see the elastic-apm-traceparent as well as the traceparent in the http request header.

And we are sure MDC is set and working because we have other fields injected in our MDC.

Is there additional setup we are missing ?

Thanks a lot for your help.
Kind Regards,
Stessy

2

Hi and welcome to the forum!

Did you set enable_log_correlation to true?

Also, did you see our log correlation documentation?

3

Hi,

yes the property is set to true.

I set the property in the domain.xml

<system-properties>
    ......
    ......
    <property name="elastic.apm.service_name" value="phenix"/>
    <property name="elastic.apm.application_packages" value="be.buyway"/>
    <property name="elastic.apm.server_urls" value="http://localhost:8200"/>
    <property name="elastic.apm.enable_log_correlation" value="true"/>
</system-properties>

And the problems remains.

If I get a copy of the MDC context map, the trace and transaction are not set.

Thanks for your help.
Stessy

4

The tracing IDs are added to the MDC when a transaction/span/error is activated and removed from MDC when they are deactivated, so only log messages written while such are active will contain them.
Try setting the log_level to debug and follow the logs, make sure a message is written to the log while there is an active event.

5

Hi,

that's really strange.

The trace and transaction can be found in our MDC only through our interceptor.
We have one interceptor around every EJB to log every access to our services. And in this interceptor, but not everytime, we can find the trace propagated from our µservice.

On the other hand, our REST endpoints don't have any trace or transaction.

I'm going to investigate a bit further.

Thanks for your help.

6

Hi,

The problem is now solved.

We have a ContainerRequestFilter which sets up and tears down some logging context properties in our MDC.

And the MDC is clear if we don't have any "Credential" in the SecurityContext instead of removing specific logging context properties.

Now by removing from our MDC only specific properties, transaction.id and trace.id are correctly set in our logs.

Sorry for the inconvenience.
Thanks for your help.
Stessy

7

Awesome, thanks for reporting about the resolution!

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.