Java Agent: Custom span in multithread context

[suikast42]

I try to catpure spans of therad cals.

For eaxample.

Thread A creates Therad B. Thread A cretes the inital span and Thread B should be listed in transaction of Thread B.

If I start the Span in Thread B with the TransactionId of Thread A then I get the folowing error:

The traceparent header has to be at least 55 chars long, but was 'bdcdf7d968cf8f7a'

What kind of ID should the ElasticApm.startTransactionWithRemoteParent(this::getID) provide ?

[felixbarny (Felix Barnsteiner)]

Could you explain your scenario in a bit more detail? Typically, you don't create a transaction in the same service when there is already one active. Have you tried starting a child span on thread B via Span#startSpan?

Does the span on thread A end before the span on thread B is started?

[suikast42]

No Thread A is not stopped unless Thread B is finished and Thread A end the inital Transaction if Thread B is finished it's work.

If I use ElasticApm.currentTransaction() in Thread B then I get a NoopTransaction. Beacause Thread B is not assoiated with a Transaction. That is expected as mentioned in the docu.

And If I use the transaction id of Thread A in Thread B to create a child span then I get the error.

My Example code in Thread B. The question is what kind of ID should getId deliver. Actualy it is the id od the inital transaction

final Transaction transaction = ElasticApm.startTransactionWithRemoteParent(this::getID)

        Span span = transaction
                .startSpan("WorkerJob", getName(), "run")
                .setName(getName())
        
        try (final Scope scope = transaction.activate()) {
            span.injectTraceHeaders((name, value) -> System.err.println("Trace headers: " + name +" -> "+value));
            Thread.sleep(random(10, 1000));
        } catch (Exception e) {
            span.captureException(e);
        } finally {
            span.end();
        }

[felixbarny (Felix Barnsteiner)]

Instead of creating another transaction, I'd suggest to transfer the transaction to thread B and to start a span from it like that:

final Transaction transaction = ElasticApm.startTransaction();
final Thread thread = new Thread(new Runnable() {
    @Override
    public void run() {
        try (Scope scope = transaction.activate()) {
            final Span span = ElasticApm.currentSpan()
                .startSpan("WorkerJob", "name", "run")
                .setName("name");
            try (Scope spanScope = span.activate()) {
                // do stuff
            } catch (Exception e) {
                span.captureException(e);
                throw e;
            } finally {
                span.end();
            }
        }
    }
});
thread.run();
thread.join();
transaction.end();

[suikast42]

Thanks for fast reply. Of course this approach works fine.

But the approach with transmit a span id is more lightweight I assume.

In my inital transaction I capture the trace ID with
transaction.injectTraceHeaders((name, value) -> traceHeader = value)
and propagate this id to Thread B. And that's works as exppected. But annoying thing here is that I need a further Transaction in Thread B which shows in the APM UI.

The unnamed Transactions are not nedded. How can I avoid that ?

[felixbarny (Felix Barnsteiner)]

@suikast42

But the approach with transmit a span id is more lightweight I assume.

Why do you think so?

@suikast42

transaction.injectTraceHeaders((name, value) -> traceHeader = value)

Currently, there is only one header but we will likely add more so you would override the traceHeader variable.

What you could do is to make a map containing the headers like that:

Map<String, String> headers = new HashMap<>();
transaction.injectTraceHeaders(headers::put);
...
Transaction transaction = ElasticApm.startTransactionWithRemoteParent(headers::get)

Just don't create spans but only use the transactions to avoid the unnamed transactions.

But I'm still not sure why you want multiple transactions. Can't you create a Transaction for Worker3 and only create Spans afterwards?

[suikast42]

@felixbarny

Why do you think so?

I have a JavaEE 8 app running in Wildfly 16. I want track the EJB boundary and the JMS onMessage Invocations with the real SQL statements. I track the real SQL with wrapping the underlying JDBC driver. For that purpose I think that's more lightweight to ship a TransactionID to the JDBC driver instead of the tranaction object.

@felixbarny

Currently, there is only one header but we will likely add more so you would override the traceHeader variable.

I got it. I see as well there's currently one header.

@felixbarny

Just don't create spans but only use the transactions to avoid the unnamed transactions.

Yes you are right. But there is a litte but big diffefence between transaction and span that I can see. Transaction doesn't report the stack.

[sabareesh1 (Sabareesh)]

Glad we are using Spring Sleuth. This looks messy

[Eyal_Koren]

@suikast42

I want track the EJB boundary and the JMS onMessage Invocations with the real SQL statements.

If you can provide a description of your app architecture, this could assist us understand what you are trying to achieve. Is it a message bean you are using? Are you monitoring both sides of it? Which one contains the sample code you enclosed? Which one of is doing JDBC calls?

I track the real SQL with wrapping the underlying JDBC driver. For that purpose I think that's more lightweight to ship a TransactionID to the JDBC driver instead of the tranaction object.

Can you further describe how you are wrapping the driver and for which purpose? Which JDBC driver are you using? Also, not sure about the performance hit you are worried about with transferring the Transaction object reference.

But there is a litte but big diffefence between transaction and span that I can see. Transaction doesn't report the stack.

Because our data model is not designed for this kind of usage (multiple transactions creating during a single request handling in the same service). This will also make your main service summary page in the APM UI less usable, as you would get an entry for each of your spans in the transaction list. If possible, @felixbarny's suggestion to activate the Transaction object on the worker thread seems most likely to provide the optimal solution.

[suikast42]

@Eyal_Koren

If you can provide a description of your app architecture, this could assist us understand what you are trying to achieve. Is it a message bean you are using? Are you monitoring both sides of it? Which one contains the sample code you enclosed? Which one of is doing JDBC calls?

The architecture relies on the standard JavaEE architecture. With some lagacy hooks of course We don't use Mdbs. We register manualy message listeners on jms ( for acivatating dan deactivating the listeners during runtime ). So JDBC calls can come from serval places . REST cal, Remote EJB, Timed actions, Event driven actions and so on.

@Eyal_Koren

Can you further describe how you are wrapping the driver and for which purpose? Which JDBC driver are you using? Also, not sure about the performance hit you are worried about with transferring the Transaction object reference.

I don't like the output of hibernate prepared statements. I can't use this queries for make performance analysis on the database. For that purpose I use DriverSpy and JSQLParser.

@Eyal_Koren

Because our data model is not designed for this kind of usage

Ok this part was my misunderstanding of spans and transactions.

[Eyal_Koren]

The injectTraceHeaders and startTransactionWithRemoteParent APIs you referred to are designed to enable manual distributed tracing, so they may be useful to assist you with the JMS and EJB monitoring.

In addition, here is an overview our data model.

I hope this helps,
Eyal.