HttpUrlConnectionInstrumentation not work when use in different thread

Elastic Observability APM
#1

Kibana version :

7.10.2

Elasticsearch version :

7.10.2

APM Server version :

1.17.0

APM Agent language and version :
Java
Browser version :
Chrome 79.0.3945.117
Original install method (e.g. download page, yum, deb, from source, etc.) and version :
download page and local setup
Fresh install or upgraded from other version?
No

**

Hello!

In our application we use code provided by third party company that handle SOAP calls.
This class AsyncResponse use HttpURLConnection to communicate. Problem is that when user click on GUI, SOAP call is creating in separated thread. Agent catch user click and create GUI transaction, but on this transaction there is no span that should be created by HttpUrlConnectionInstrumentation. When SOAP call is creating in same thread that GUI all work fine. I get span created by HttpUrlConnectionInstrumentation.

Class is big and I can not provide code so I will show some pseudo code to better understand

public class AsyncResponse {

    private AsyncResponse<T, E>.WebserviceThread _thread = new AsyncResponse.WebserviceThread();

    private class WebserviceThread extends Thread {
        WebserviceInvocationThread() {
            super("WSI-Invocation");
        }

        public void run() {
            // connect HttpURLConnection
            // inputStream
            // outputStream
            // here should create Span from HttpUrlConnectionInstrumentation
        }

        public void waitForCallComplete(long timeout) {
            // check timeout and throw TimeoutException
        }
    }

    public void getResponse() {
        this._thread.start();
        this.waitForCallComplete(90000);
    }
}

Seem to when creating new child thread all APM context is loosing.
In test package I see code that test HttpUrlConnectionInstrumentation in differen thread.

public void testEndInDifferentThread() throws Exception {
    final HttpURLConnection urlConnection = new HttpURLConnectionWrapper(new HttpURLConnectionWrapper((HttpURLConnection) new URL(getBaseUrl() + "/").openConnection()));
    urlConnection.connect();
    final Thread thread = new Thread(tracer.getActive().withActive(() -> {
        try {
            urlConnection.getInputStream();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }));
    thread.start();
    thread.join();

    verifyHttpSpan("/");
}

Problem Is that I can not change code where different thread is created.
Any idea how to solve this issue ?
Thank you for help

#2

Thanks for the question.

It is exactly it.

Creating a new thread for a frequent task like SOAP request is very uncommon, therefore we do not support that and are unlikely to add support for that soon, unless we get more indications it is needed. Normally, such tasks are handled through thread pools and we support a lot of those through the instrumentation of inherent java.util.concurrent APIs.

See if your SOAP client offers some sort of interceptor concept that allows you to provide callbacks that will be called before and after the request is made. If you have this option, you may do it manually: find a place in your code where you can get the active span through our API and add it to the request metadata. Then, when your interceptor's beforeRequest gets invoked, activate the span and keep the returned Scope and when the afterRequest is called, close the scope.
Otherwise, you may try to write an external plugin that does something similar to what we do with our java-concurrent plugin, but be aware that this may be complicated to get right.

#3

Thank you for help.
Class interceptors are not aviable :frowning:

Own plugin link java-concurrent plugin is last solution.

I think about change inner Thread object in runtime. What do you think ? It is possible by using APM agent ?

public class AsyncResponse {
// Inject something else with APM context
private AsyncResponse<T, E>.WebserviceThread _thread = new AsyncResponse.WebserviceThread();

private class WebserviceThread extends Thread {
}
}
#4

I wouldn't recommend messing with the filed assignment - the low-level bytecode instrumentation library (ASM) certainly has such capabilities, but we are using a higher-level library on top of that (Byte Buddy) in which these type of manipulations are not the norm. It would require very delicate bytecode handling and you will need to guard from type mismatches.

If you go for a plugin, I would recommend something like that: like in our concurrent plugin, rely on a mapping between the context (parent span) and the thread that runs the task. I'd focus at the instrumentation of WebserviceThread#getResponse method, because it has access to both - it is still executed on the traced thread, so you can get the active span through currentSpan and the designated thread through the _thread field. So, it would look something like:

@Advice.OnMethodEnter(suppress = Throwable.class, inline = false)
public static void onBeforeGetResponse(@Advice.FieldValue("_thread") @Nullable Object webServiceThread) {
    Span span = ElasticApm.currentSpan();
    // map webServiceThread to span in a map produced by co.elastic.apm.agent.sdk.weakmap.WeakMapSupplier
    thread2spanMap.put(webServiceThread, span);
}

Then, you can instrument WebserviceThread#run as follows:

@Advice.OnMethodEnter(suppress = Throwable.class, inline = false)
public static Object onBeforeGetResponse(@Advice.This Objet thisThread) {
    Span span = thread2spanMap.remove(thisThread)
    if (span != null) {
         return span.activate();
    }
}

@Advice.OnMethodExit(suppress = Throwable.class, onThrowable = Throwable.class, inline = false)
public static void onExitServletService(@Advice.Enter @Nullable Object scope) {
    ((co.elastic.apm.api.Scope) scope).close();
}

This is of course a rough suggestion, so don't assume it will just work as is, but I hope it can give you a direction.

1 Like
#5

Greate suggestion. Thank you.

We use APM in version 1.17.0. This class had to be added in later version.
In 1.17.0 java-concurrent-plugin plugin use:
public static final WeakConcurrentSet<Executor> excluded = DataStructures.createWeakConcurrentSetWithCleanerThread();

WeakConcurrentSet do same job ? :slight_smile:

#6

We did a major change in 1.18.0, part of which was to ease the creation of community plugins, including the entire sdk and external plugin concept. So it is highly advised not to write a plugin for older versions, but instead upgrade to latest and start there. It should be a fairly easy upgrade, and you'll get additional features.

#7

Sure, upgrade is always something nice to see. Unfortunately not easy task in my situation.
Problem is that we use very specific platform. I can't change code of this platform. To make APM Agent work some team members made change in core APM code. I know this is wrong way, but it was the only way to make APM work.

I will consider update to 1.21.0.

In the meantime, my question about WeakConcurrentSet and WeakMapSupplier is still open :wink:
Thank you!

#8

You can definitely use WeakConcurrentMap (why set?).

But if your team can transfer anything you already did to an external plugin, instead of a using a fork, it will provide you exactly this decoupling and things are likely to stay much more stable in future releases, allowing you to upgrade whenever you need.

#9

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.