Java Agent: How to manually instrument an RPC framework which is not already supported?

[oldcodeoberyn]

I want to instrument dubbo which is the popular microservice framework in China, it's default RPC framework is not base on http. For each remote call, this framework will exec a filter chain of which I am going to put the instrument code in.

I am tring to use the public API.
for the consumer filter, I sent the transaction id to provider:

    @Override
    public Result invoke(
                    Invoker<?> invoker, Invocation invocation ) throws RpcException
    {
        Transaction transaction = ElasticApm.startTransaction();
        try (final Scope scope = transaction.activate())
        {
            // this prc context will send to provider
            RpcContext.getContext().setAttachment("parentId", transaction.getId());
            String name = invocation.getMethodName();
            transaction.setName( name );
            transaction.setType( Transaction.TYPE_REQUEST );
            return invoker.invoke(invocation);
        }
        catch( Exception e )
        {
            transaction.captureException( e );
            throw e;
        }
        finally
        {
            transaction.end();
        }
    }

for the provider filter, I use the consumer transaction id to create transaction with parent:

    @Override
    public Result invoke(
                    Invoker<?> invoker, Invocation invocation ) throws RpcException
    {
        // use startTransactionWithRemoteParent to create transaction with parent, which id from prc context
        Transaction transaction = ElasticApm.startTransactionWithRemoteParent(key -> invocation.getAttachment( "parentId" ));

        try (final Scope scope = transaction.activate())
        {
            String[] interfaceArr = invocation.getAttachment( "interface" ).split( "\\." );
            String className = interfaceArr[interfaceArr.length - 1];
            String name = className + "#" + invocation.getMethodName();
            transaction.setName( name );
            transaction.setType( Transaction.TYPE_REQUEST );
            return invoker.invoke(invocation);
        }
        catch( Exception e )
        {
            transaction.captureException( e );
            throw e;
        }
        finally
        {
            transaction.end();
        }
    }

but after I run it, the provider transaction didn't correct set the parent id. actually, in the public API, it do not have clear description about how the build a child-parent relationship between transactions of RPC

Could you please give me a hint about how to do it with the public API? espacially the span.injectTraceHeaders(request::addHeader), I don't know how to use it, Thanks in advance!

[felixbarny (Felix Barnsteiner)]

Hi and thanks for your question. I hope this example helps:

@Override
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
    Span span = ElasticApm.currentTransaction()
            .startSpan("external", "http", null)
            .setName(invocation.getMethodName());
    try (final Scope scope = transaction.activate()) {
        span.injectTraceHeaders((name, value) -> RpcContext.getContext().setAttachment(name, value));
        return invoker.invoke(invocation);
    } catch (Exception e) {
        span.captureException(e);
        throw e;
    } finally {
        span.end();
    }
}

@Override
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
    // use startTransactionWithRemoteParent to create transaction with parent, which id from prc context
    Transaction transaction = ElasticApm.startTransactionWithRemoteParent(key -> invocation.getAttachment(key));

    try (final Scope scope = transaction.activate()) {
        String[] interfaceArr = invocation.getAttachment("interface").split("\\.");
        String className = interfaceArr[interfaceArr.length - 1];
        String name = className + "#" + invocation.getMethodName();
        transaction.setName(name);
        transaction.setType(Transaction.TYPE_REQUEST);
        return invoker.invoke(invocation);
    } catch (Exception e) {
        transaction.captureException(e);
        throw e;
    } finally {
        transaction.end();
    }
}

If that works for you, I'll add a simplified version of this to the documentation.

Cheers,
Felix

[felixbarny (Felix Barnsteiner)]

I have added improvements to the documentation: https://github.com/elastic/apm-agent-java/pull/529

[oldcodeoberyn]

Hi felixbarny, thanks it work. And my next step is going to make it become formal support by elastic APM, I already submit the survey about wanted support technology for dubbo [dubbo support the RPC protocol of dubbo, pb, thrift, hessian, etc, but with the filter, we could handle it with one code], but don't know whether you will accept it and what is the plan. If I want to do it myself, Is there any API level document about how to make it by ElasticApmApiInstrumentation? I know there have a cook-book, but for me, it is not enough:sweat_smile:

[felixbarny (Felix Barnsteiner)]

Yay, very cool

We would very happily accept a PR adding support for dubbo. We have some more contributing information here: https://github.com/elastic/apm-agent-java/blob/master/CONTRIBUTING.md and here: https://github.com/elastic/apm-agent-java/blob/master/apm-agent-plugins/README.md

You can also look at how other plugins are implemented. For example this one: https://github.com/elastic/apm-agent-java/blob/master/apm-agent-plugins/apm-spring-resttemplate-plugin/src/main/java/co/elastic/apm/agent/resttemplate/SpringRestTemplateInstrumentation.java

A good way to start would be to implement some tests. That makes it easier for us to help you with the implementation of the actual instrumentation.

Just start with something and we will be there to help you

--
Felix