How to use Distributed tracing in with elastic apm in non .net core application?

[sonammarda]

Kibana version: 7.3

Elasticsearch version: 7.3

APM Server version: 7.2

APM Agent language and version: Using for .NET project.

My project is in .Net Framework 4.6.1. I can capture transactions now in my project. Now I want to add distributed tracing and log all SQL queries executed in calls. I have used below code for distributed tracing:

var tracingData = Elastic.Apm.Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData.SerializeToString();

Elastic.Apm.Agent.Tracer.CaptureTransaction(transactionName, Elastic.Apm.Api.ApiConstants.TypeRequest, transaction =>
{
transaction.Tags.Add("UserId", UserId.ToString());
transaction.Tags.Add("EntityId", EntityId.ToString());
transaction.Tags.Add("UserName", UserName);
}, DistributedTracingData.TryDeserializeFromString(tracingData));

However I do not get any SQL queries or detailed traces of my call. My project structure is:

1. API Layer
2. Service/Application layer
3. Repository Layer(Which has linq queries or calls to Stored Procedure)

I have added above code to start and capture transaction and distributed tracing in API layer. So now what more changes I have to make to get all SQL queries executed and distributed tracing.

Below is what I get currently-

[Sergey_Kleyman]

Hi @sonammarda,

First I would recommend to use the latest version of the agent (which is 1.0.1 at the moment). Please note that we renamed Tags to Labels before 1.0 release.

Regarding the code you have provided - you do run

var tracingData = Elastic.Apm.Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData.SerializeToString();

on the client side of your service communication (for example on API layer in the case of the first leg of communication between the layers of your application) and

Elastic.Apm.Agent.Tracer.CaptureTransaction(transactionName, Elastic.Apm.Api.ApiConstants.TypeRequest, transaction =>

on the server side of the of your service communication (for example on Service/Application layer in the case of the first leg of communication between the layers of your application), right?

[Sergey_Kleyman]

An additional question - which one is true in your case:

1. You see transactions with SQL queries but they are not connected with other layers into one trace
2. You don't see transactions with SQL queries

If it's (2) then the follow up question - have you added code like the one you provided:

Elastic.Apm.Agent.Tracer.CaptureTransaction(transactionName, Elastic.Apm.Api.ApiConstants.TypeRequest, transaction =>

to Repository Layer(Which has linq queries or calls to Stored Procedure) service's code?

[sonammarda]

Second one is true in my case. I do not see any SQL queries. I have written an attribute in web api to start and capture transaction. Below is my code:

public override void OnActionExecuting(HttpActionContext actionContext)
{
Elastic.Apm.Agent.Tracer.StartTransaction(someTransactionName, Elastic.Apm.Api.ApiConstants.TypeRequest);
}

public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
{
var tracingData = Elastic.Apm.Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData.SerializeToString();
if (actionExecutedContext.Exception != null)
{ Elastic.Apm.Agent.Tracer.CurrentTransaction?.CaptureException(actionExecutedContext.Exception, apiContextInfo, false, transactionName);
}

        Elastic.Apm.Agent.Tracer.CaptureTransaction(transactionName, Elastic.Apm.Api.ApiConstants.TypeRequest,
           transaction =>
           {
               transaction.Tags.Add("UserId", apiContext?.UserId.ToString());
               transaction.Tags.Add("EntityId", apiContext?.EntityId.ToString());
               transaction.Tags.Add("UserName", apiContext?.UserName);
           }, DistributedTracingData.TryDeserializeFromString(tracingData));
    }

Both the above calls are made in web api layer only.

I also tried to start and capture transaction with distributed tracing data in Repository layer(where I have my linq and call to SP). Below is the code for that-

public CartSummaryDetailsDTO GetCartSummaryDetails(int userId)
{
Elastic.Apm.Agent.Tracer.StartTransaction("GetCartSummaryDetails-CartRepo Call", Elastic.Apm.Api.ApiConstants.TypeRequest);

        //My project Linq code here

Elastic.Apm.Agent.Tracer.CaptureTransaction("GetCartSummaryDetails-CartRepo Call", Elastic.Apm.Api.ApiConstants.TypeRequest,
transaction =>
{
transaction.Tags.Add("UserId", "someid");
transaction.Tags.Add("EntityId", "someentityid");
transaction.Tags.Add("UserName", "someusername");
}, DistributedTracingData.TryDeserializeFromString(Elastic.Apm.Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData.SerializeToString()));
}

But with both the above methods, I am not able to see any SQL queries in kibana.
I can only see the transaction captured in kibana with the transaction name that I have mentioned in code and added tags are seen under metadata. But no SQL queries.

Screenshot of what I see-

Appreciate any help!

[Sergey_Kleyman]

The reason for SQL queries not being captured is because Elastic APM .NET agent doesn't have support for automatically capturing LINQ to SQL queries. As you can see in Supported Technologies section of the documentation the only data access technology for which .NET agent currently supports automatic instrumentation is Entity Framework (EF) Core.

I didn't see that this is the actual reason for SQL queries not being captured - I focused on distributed tracing aspect of the question.

I opened an issue to add automatic instrumentation for LINQ to SQL. Although it's quite dated technology so I'm not sure if we will get to it in the near future. Of course community contributions are very welcome in case you would like to give a try. I added a few potential directions for implementation in the issue's description.

[sonammarda]

Thanks a lot for all your help!

I have one more query. As I mentioned earlier, I am able to see traces to my application(API calls) and exceptions raised in my application.

While capturing transaction, I am able to add tags which can be seen in metadata-
Elastic.Apm.Agent.Tracer.CaptureTransaction(transactionName, Elastic.Apm.Api.ApiConstants.TypeRequest,
transaction =>
{
transaction.Tags.Add("UserId", apiContext?.UserId.ToString());
transaction.Tags.Add("EntityId", apiContext?.EntityId.ToString());
transaction.Tags.Add("UserName", apiContext?.UserName);
}, DistributedTracingData.TryDeserializeFromString(tracingData));

But while capturing exception, I do not see any option to add tags to be seen in metadata. I want to add additional tags/labels under metadata while capturing exception . Please help me to do this.

[Sergey_Kleyman]

Unfortunately providing additional labels in CaptureError and CaptureException agent's public API is not supported by .NET agent yet. I've opened an issue for this enhancement.

The only workaround I can suggest until we implement the above enhancement is to add error specific labels to the transaction using Labels API - the labels will be inherited by the error events generated by subsequent CaptureError and CaptureException calls because error events inherit their context (which in particular includes labels) from the corresponding transaction.