End to end tracing is not shown

feroz-ahmmed · June 20, 2021, 12:11pm

Kibana version: 7.11.2

Elasticsearch version: 7.11.2

APM Server version: 7.11.2

APM Agent language and version: Elastic.Apm.AspNetCore@1.10.0

Browser version: Chrome 91.0.4472.77

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Fresh install

Is there anything special in your setup?

Default configurations.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have two dotnet webapi services. Service1 calls Service2 via REST using the default HttpClient. Both of the requests to individual services can be found from APM section of kibana. However, full trace is not shown for the first request. In other words, I can not see that Service1 called Service2 in the request lifecycle.

I am expecting the end to end trace of the original request like this one -

Steps to reproduce:

Clone the git repository GitHub - feroz-ahmmed/elastic-apm-demo
Set ElasticApm:ServerUrl and ElasticApm:SecretToken in the appsettings.json for both Service1 and Service2.
Run both services.
Send a Get request on /Service1 endpoint of Service1.
Find the request in APM section of kibana.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

feroz-ahmmed · July 8, 2021, 10:30am

Replying so that this thread doesn't get closed.

forloop · July 20, 2021, 12:37am

Hi @feroz-ahmmed,

There's a couple of things that need to be addressed for this to work:

The service name for TracingService2 should be TracingService2 (or something else, but different from TracingService1). It's set to the same service name as TracingService1 in the example:

github.com

feroz-ahmmed/elastic-apm-demo/blob/48f0cf4780cdac458e033fc937da6056484a8490/TracingService2/Service2.Api/appsettings.json#L10

    
      
          {
            "Logging": {
              "LogLevel": {
                "Default": "Information",
                "Microsoft": "Warning",
                "Microsoft.Hosting.Lifetime": "Information"
              }
            },
            "ElasticApm": {
              "ServiceName": "TracingService1",
              "SecretToken": "<secret-token>",
              "CloudProvider": "none",
              "Environment": "Local",
              "LogLevel": "Warning",
              "ServerUrl": "<apm-url>"
            },
            "AllowedHosts": "*"
          }

The APM agent is not configured to trace HTTP calls. The HttpDiagnosticsSubscriber needs to passed to app.UseElasticApm(Configuration, <subscribers here>) in at least TracingService1, though you probably want to trace HTTP calls in both services, so add it to both in

github.com

feroz-ahmmed/elastic-apm-demo/blob/48f0cf4780cdac458e033fc937da6056484a8490/TracingService1/Service1.Api/Startup.cs#L33

    
      
              services.AddControllers();
              services.AddSwaggerGen(c =>
              {
                  c.SwaggerDoc("v1", new OpenApiInfo {Title = "WebApplication", Version = "v1"});
              });
          }
          
          
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
          public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
          {
              app.UseElasticApm(Configuration);
              
              if (env.IsDevelopment())
              {
                  app.UseDeveloperExceptionPage();
                  app.UseSwagger();
                  app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "WebApplication v1"));
              }
          
          
    app.UseRouting();

and

github.com

feroz-ahmmed/elastic-apm-demo/blob/48f0cf4780cdac458e033fc937da6056484a8490/TracingService2/Service2.Api/Startup.cs#L47

    
      
                  // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
                  public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
                  {
                      if (env.IsDevelopment())
                      {
                          app.UseDeveloperExceptionPage();
                          app.UseSwagger();
                          app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "Service2.Api v1"));
                      }
          
          
            app.UseElasticApm(Configuration);
          
          
            app.UseRouting();
          
          
            app.UseAuthorization();
          
          
            app.UseEndpoints(endpoints => { endpoints.MapControllers(); });
                  }
              }
          }

i.e.

app.UseElasticApm(Configuration, new HttpDiagnosticsSubscriber());

With these changes, distributed tracing works

If you want to let the APM agent configure all subscribers, you can reference the Elastic.Apm.NetCoreAll package, and call

app.UseAllElasticApm(Configuration);

which will subcribe HttpDiagnosticsSubscriber along with many others (see docs). For 1.11.0, these are

github.com

elastic/apm-agent-dotnet/blob/1fb983d5a9a18f045f0c7c5d275f626ca6cd77a0/src/Elastic.Apm.NetCoreAll/ApmMiddlewareExtension.cs#L46-L57

    
      
          			.UseElasticApm(builder, configuration,
          				new HttpDiagnosticsSubscriber(),
          				new EfCoreDiagnosticsSubscriber(),
          				new SqlClientDiagnosticSubscriber(),
          				new ElasticsearchDiagnosticsSubscriber(),
          				new GrpcClientDiagnosticSubscriber(),
          				new AzureMessagingServiceBusDiagnosticsSubscriber(),
          				new MicrosoftAzureServiceBusDiagnosticsSubscriber(),
          				new AzureBlobStorageDiagnosticsSubscriber(),
          				new AzureQueueStorageDiagnosticsSubscriber(),
          				new AzureFileShareStorageDiagnosticsSubscriber(),
          				new MongoDbDiagnosticsSubscriber());

feroz-ahmmed · July 25, 2021, 11:49am

HI @forloop ,
Sorry for late reply. I tried your proposed changes changes and now I can see the end to end request trace.

Thanks for your time and detailed reply.

system · August 15, 2021, 7:50am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
APM Traces Tab Doesn't Split out different services APM ui	3	603	January 23, 2020
APM trace view in Kibana displays the traced services as external services when they are called via the API gateway APM java	5	174	June 28, 2024
Service Map not linking services APM dotnet	5	1898	February 27, 2021
Distributed Tracing on .Net FullFramework APM dotnet	3	826	May 11, 2020
Distributed tracing missing expansion for one service APM ruby	5	727	October 21, 2019

End to end tracing is not shown

Related topics