ELK APM Transaction not showing end to end flow if we have two diffent services

Vissu_Reddy · May 19, 2020, 4:39am

We have two services called ServiceA and ServiceB . ServiceA has been implemented using Java language and ServiceB has been implemented using .Net language. Both these services has been integrated with respective ELK APM agents. Also we instrumented code both these services using Agents and were able to see ELK APM metrics separately in APM UI in Kibana but not in Single transaction view

Here is Question.

We are calling ServiceB from ServiceA , We were able to see ELK APM metrics separately in APM UI in Kibana not in Single transaction View.

For ServiceA apm metrics are showing under ServiceA and for ServiceB apm metrics are showing under ServiceB in ELK APM UI. It’s not showing end to end flow in single transaction. It’s very hard to get mapping from ServiceA to ServiceB since both of these metrics are not in single transaction. If I want to get mapping between ServiceA and ServiceB, manually I have to verify for ServiceA and ServiceB apm metrics

Is there is way to display ServiceA and ServiceB apm metrics in single view Transaction? Is there is way to display end to end flow in single view Transaction?

nugusbayevkk · May 19, 2020, 4:58am

Hi,
What do you mean by "metrics" ?
Can you share screenshot?

Vissu_Reddy · May 19, 2020, 5:11am

I mean Transaction.. while we are calling from ServiceB from ServiceA, there is one transaction has been created for ServiceA and another transaction has been created for ServiceB. we were able to see two diffent transactions has been created in ELK APM UI. is there is way to put these two transactions into one transaction so that we can easy to map it down

nugusbayevkk · May 19, 2020, 5:23am

It's about distribute tracing, if you use supported technologies, you should see all information in transaction details of ServiceB.
Can you share screenshot from ServiceB transaction details?

Vissu_Reddy · May 19, 2020, 5:37am

I can’t share screen shot as its company proprietary

As I said, ServiceA implemented using Java language and integrated with ELK Java APM agent and ServiceB implemented using .Net language and integrated with ELK .Net APM agent

If we see distributed tracing there were two separated transactions are created one for ServiceA and another for ServiceB

In ServiceA transaction details I don’t see ServiceB information details

I just all about to know is there is way to combine these two transactions into one using distributed tracing?

nugusbayevkk · May 19, 2020, 5:41am

Ok, thanks.
How does serviceA communicate with serviceB? What technology do you use?

Vissu_Reddy · May 19, 2020, 6:31am

Its just a another api ... We are calling serviceB api in ServiceA.

ServiceA - Java
ServiceB - .Net

can we connect thru bridge is it posible ? so that I'll show you my usecase easy for undestand.

erjohnson · May 21, 2020, 9:33pm

I have this issue as well, though both of my applications are .NET 4.8. I set up out-of-the-box MVC ("Web") and Web API ("API") applications, added the Elastic.Apm.AspNetFullFramework 1.5.1 NuGet package, and configured them to talk to our APM Server. Both APM Server and Kibana are 7.6.1.

I then added a call to the API from the MVC app. They are both recording their individual transactions correctly, and the transactions are linked in the index by a shared trace ID, but Kibana only shows the Web transaction for the trace:

erjohnson · May 21, 2020, 10:26pm

The exact same setup, except using .NET Core apps, works fine:

Is there some other metadata, beyond the trace ID, necessary to create the distributed trace waterfall which the FullFramework client library isn't populating?

erjohnson · May 21, 2020, 11:03pm

After some more digging, this appears to either be an issue with HttpDiagnosticsSubscriber or the way I'm using it. Notice that in the Core screenshot, there is an intermediate span for the HTTP request from the MVC app to the API app, while the only two spans associated with the trace in the Full Framework screenshot are the respective transactions. Indeed, the parent ID of the API transaction is not the Web transaction:

Furthermore, there is no span with that ID:

As expected, in the Core setup the parent ID of the API transaction corresponds to the Web->API HTTP request span:

I had tried to add auto-instrumentation to HttpClient in the Full Framework apps with this code in Global.asax.cs:

private static volatile bool apmInitialized;

public override void Init()
{
    if (!apmInitialized && Agent.IsConfigured)
    {
        apmInitialized = true;
        Agent.Subscribe(new HttpDiagnosticsSubscriber());
    }
}

Is that incorrect? I did verify that Agent.Subscribe is being called as expected, and before the outgoing HTTP request is sent. Even when commented out in the Web app, the API transaction is associated with a nonexistent parent ID.

erjohnson · May 22, 2020, 11:16pm

After spending a lot of time with the AspNetFullFrameworkSampleApp sample code, I figured out the issue.

First of all, HttpDiagnosticsSubscriber is enabled by default in the Full Framework package, in the HTTP module:

private static bool InitOnceForAllInstancesUnderLock(string dbgInstanceName) =>
	InitOnceHelper.IfNotInited?.Init(() =>
	{
		SafeAgentSetup(dbgInstanceName);

		_isCaptureHeadersEnabled = Agent.Instance.ConfigurationReader.CaptureHeaders;

		Agent.Instance.Subscribe(new HttpDiagnosticsSubscriber());
	}) ?? false;

I think this is different from how the Core packages & API behave, where you have to explicitly enable anything other the ASP.NET instrumentation.

Second, even though it's enabled by default, the spans it creates won't actually be instrumented and captured unless the Microsoft.ApplicationInsights.Web package & module are installed.

Once that module is added, the HTTP client spans are captured as expected, and the API application transaction can be linked to the Web application transaction, creating a distributed trace as expected.

I don't know if either of those things are documented anywhere, but if so I'd appreciate a link. This was a confusing rabbit hole to go down, compared to the ease of the Core integration.

GregKalapos · May 23, 2020, 11:49am

Hi @erjohnson,

I went through the comments here - couple of comments:

On the issue on .NET Framework that no span is created for outgoing HTTP requests:

We had a similar issue here and the problem was that the application referenced an older System.Diagnostics.DiagnosticSource. The HttpDiagnosticsSubscriber just subscribes to DiagnosticSource events, but that old version does not send the necessary events. I suspect that by adding Microsoft.ApplicationInsights.Web you pulled a newer version of that package and that's why it started to work. The Microsoft.ApplicationInsights.Web itself is definitely not used by the Elastic agent. I may be wrong on the root cause here - you can simply double check by just updating System.Diagnostics.DiagnosticSource as described in the other thread and see if that helps.
About the difference between the Core and the non-Core packages:

This part of the doc describes what these packages do. For .NET Core the Elastic.Apm.NetCoreAll turns on everything that can be monitored on .NET Core, so if you use that, that'll automatically turn on the HttpDiagnosticsSubscriber and other things. The Elastic.Apm.AspNetCore package only turns on ASP.NET Core incoming monitoring. The idea here is that if you just want to turn on the agent use the Elastic.Apm.NetCoreAll - that'll turn on everything, but it may pull in some references that are not even needed (e.g. it'll try to turn on EF Core monitoring and and pull in some references no matter if you use EF Core or not). If this is an issue for you and want to minimize dependencies then you can use specific packages and turn on whatever you need with Agent.Subscribe manually.
We plan to have a similar "turn on everything for me, I don't care about references" type of package for .NET Framework later as well. Currently the IIS module from Elastic.Apm.AspNetFullFrameworkturns on outgoing HTTP monitoring as you pointed out.

erjohnson · May 26, 2020, 7:36pm

Thanks, @GregKalapos. After looking at the linked thread and @rajivharris's repo, I think there is another issue. Your solution worked for Elastic.Apm.* 1.4.0, but after upgrading to 1.5.x, even with the most current release version of the System.Diagnostics.DiagnosticSource package (4.7.1, assembly version 4.0.5.0), the HttpClient spans are again not created.

Can you confirm that you can reproduce my issue simply by upgrading one of his sample projects to the 1.5.0 or 1.5.1 Elastic.Apm.* packages?

Updating to the latest prelease version of System.Diagnostics.DiagnosticSource, 5.0.0-preview.4.20251.6, fixes the issue in both 1.5.0 and 1.5.1.

system · June 16, 2020, 3:36pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
End to end tracing is not shown APM dotnet	4	1102	August 15, 2021
Getting Transaction Name as 1#run only APM java	6	272	April 4, 2024
Java APM agent integration is not showing external request/transaction data APM elastic-stack-monitoring , java	7	29	September 2, 2024
Java Application REST Transactions are not showing in kibana,Elastic APM APM java	3	29	September 18, 2024
Remaining Transactions! APM nodejs , ui	4	116	August 7, 2024

ELK APM Transaction not showing end to end flow if we have two diffent services

Related Topics