Java self-attach agent + public API - How to properly pass the scope and close span-children without closing parent-span

Hi ,

Kibana version : 7.4.0 , running on ECE
Elasticsearch version : 7.4.0 , running on ECE
APM Server version : 7.4.0 , running on ECE
APM Agent language and version : Java - 1.11.0 - standalone attach

I have a proprietary, OSGI based backend application that compiles into java in the background. (SoftwareAG webMethods)
Within this backend application, i can call custom java snippets where needed.
The backend application uses several springboot microservices which are also monitored using APM.

I'm trying to manually instrument APM transactions and spans within the backend application, however I'm struggling a bit with the documentation on how to activate/deactivate spans before creating children.

Agent is set to not instrument automatically since this doesn't really result in valid spans due to the way the application works.

Bear with me, this might take some explaining...

My test case should end up looking like this:

Parent Transaction1
--- Child Span 1
---------GrandChild Span 2
--- Child Span 3
---------GrandChild Span 4

Or in apm timeline format:

--------------------------parent-----------------------
  --------child1--------
     --grandchild2--
                                 --------child1--------
                                    --grandchild2--

I've created seperate snippets to create/close transactions and spans.
All spans are created within the same thread, java objects created in earlier snippets are accessible, but not easily passable throughout the code (so not as an input to the snippet).

Start transaction:

Transaction transaction = ElasticApm.startTransactionWithRemoteParent(key -> MessageID).setStartTimestamp(unixtime);
transaction.activate();
transaction.setName(service);
transaction.setType(Transaction.TYPE_REQUEST);

Stop transaction:

Transaction transaction = ElasticApm.currentTransaction();
long unixtime = timeStampToEpoch(stopTimeStamp);
transaction.setResult(status);
transaction.end(unixtime);

Start Span: --> I image I need to do something here to check if there is a possible parent span that's still active.

Transaction transaction = ElasticApm.currentTransaction();
Span span = transaction.startSpan("webmethods", "flow", "invoke");
Scope scope = span.activate();
scope.end();	

Stop Span:

Span span = ElasticApm.currentSpan();
Scope scope = span.activate();	
span.end(unixtime);
scope.end();	

On the higher level, i'm then calling my snippets as in this pseudo code:
If this where normal java code, I image it would look more or less the same:

Transaction1:

StartTransaction()
Child1()
Otherfunctionalities()
Child2()
StopTransaction()

Children:

StartSpan()
GrandChild()
Otherfunctionalities()
StopSpan() 

Grandchildren:

StartSpan()
GrandChild()
StopSpan() 

Now, ...
this works without throwing any errors; however it seems my spans are not gettign created correctly.

What I see in APM is:

Parent Transaction1
---------GrandChild Span 2
---------GrandChild Span 4

Furthermore,
Span 2 has the starttime of what was supposed to be Span 1 and Span 4 has the starttime of Span 3.

The same goes for simpler use cases where I call the snippets in this order:

Parent.startSpan()
Child.startSpan()
Child.endSpan() --> actually ends parent
Parent.endSpan() --> actually ends child

This results in my child having a later endtime than my parent.

So my question is:
If this where normal Java, how should I be manually creating and ending scopes while calling nested functions, to prevent a child function from picking up and ending the parent transaction?

In general, you should also ways activate within a try/finally. See also https://www.elastic.co/guide/en/apm/agent/java/current/public-api.html#api-span-activate

It seems you are activating your transaction but not deactivating it and you activate your span twice. Seems like that's the problem here. If you forget to deactivate (aka leak a scope) bad things can happen.

Hi Felix,

So, should I deactivate my transaction before entering the child method?
And reactivate it when exiting the child method?

Same for when I enter a child-function from within a span?

The docs are rather clear on how to create single spans within a function, but i'm not entirely following how i can make sure that a newly created span for a child-function doesn't activate the parent-span.

Something similar to this should work:

public void parentMethod() {
    Transaction transaction = ElasticApm.startTransactionWithRemoteParent(key -> MessageID).setStartTimestamp(unixtime);
    try (final Scope scope = transaction.activate()) {
        transaction.setName("transaction name");
        transaction.setType(Transaction.TYPE_REQUEST);
        childMethod();
    } catch (Exception e) {
        transaction.captureException(e);
        throw e;
    } finally {
        long unixtime = timeStampToEpoch(stopTimeStamp);
        transaction.setResult(status);
        transaction.end(unixtime);
    }
}

public void childMethod() {
    Span span = ElasticApm.currentSpan().startSpan("webmethods", "flow", "invoke");
    span.setName("span name");
    try (final Scope scope = span.activate()) {
        //
    } catch (Exception e) {
        span.captureException(e);
        throw e;
    } finally {
        span.end(unixtime);
    }
}