OpenTelemetry agent ASP.NET export to APM-server not working

[michael6 (michael)]

I'm trying to use OpenTelemetry to instrument a ASP.NET app and send the data to Elastic Cloud via the APM-server that comes with it. However I can't get the OpenTelemetry agent to send the data to elastic.

Kibana version: 7.17

Elasticsearch version: 7.17

APM Server version: 7.17

APM Agent language and version: OpenTelemetry 1.3.0 .NET Nuget package

Browser version: Not relevant

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Cloud

Fresh install or upgraded from other version? fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
Nothing special just an APM-server as part of Elastic Cloud. OpenTelemetry agent in example code on local machine

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
OpenTelemetry agent should be able to send data to APM-server using the OpenTelemetry protocol. Data is not showing up in apm dashboard in Kibana.

Steps to reproduce:
App used for testing is a basic ASP.NET Core API demo the agent implementation in Program.cs is below

using OpenTelemetry.Exporter;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;


var serviceName = "myTestService";
var serviceVersion = "1.0.0";

var builder = WebApplication.CreateBuilder(args);

// Configure to send data via the OTLP exporter.
builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder =>
{
    tracerProviderBuilder
    .AddConsoleExporter()
    .AddOtlpExporter(opt =>
    {
        opt.Endpoint = new Uri("https://xxxxxxxxxxxxxxxxxxxxxx:443");
        opt.Protocol = OtlpExportProtocol.Grpc;
        opt.Headers = "Authorization=Bearer ********";
    })
    .AddSource(serviceName)
    .SetResourceBuilder(
        ResourceBuilder.CreateDefault()
            .AddService(serviceName: serviceName, serviceVersion: serviceVersion))
    .AddHttpClientInstrumentation()
    .AddAspNetCoreInstrumentation();
});

// Add services to the container.

builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

I've tried switch the protocol from Grpc to HttpProtobuf but no luck there.
I can see that the instrumentation is working via the console exporter, so it seems that the problem is in de configuration of the connection to the apm-server.

[GregKalapos]

Hi @michael6 ,

opt.Endpoint = new Uri("https://xxxxxxxxxxxxxxxxxxxxxx:443");

Have you tried port 8200?

OpenTelemetry integration | APM User Guide [8.2] | Elastic

[michael6 (michael)]

Hi @GregKalapos,

thank you for the quick reply, I did try and it didn't work either.
I left it on this port as it's the one provided by the elastic cloud management page.

[GregKalapos]

Hmm,

then I'd try to debug the OpenTelemetry library. You can collect debug info about it via EventSource. More info here.

[michael6 (michael)]

I've enabled the EventSource and when I use the Grpc protocol I get errors like

{Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.", DebugException="System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.

but when I use the HttpProtobuf protocol I see no errors which suggests to me that it can connect using the HttpProtobuf protocol. I'm just not seeing anything show up in kibana so it might be something in the processing inside of the APM-server or the indexing into Elasticsearch.
I'm going to see I can get some more extra info by using a local APM-server, will update if I find something.

[GregKalapos]

I'm going to see I can get some more extra info by using a local APM-server

That's a good idea.
I'd also look at APM Server logs. In case you haven't seen it yet, log related configs that can be useful in this case, are documented here: Configure logging | APM Server Reference [7.15] | Elastic

[michael6 (michael)]

Using the local APM-server I found out that de agent connect over both http and grpc to the apm-server as shown in the logs

{"log.level":"info","@timestamp":"2022-06-15T11:24:56.203Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"POST","user_agent.original":"","source.address":"xxxx","http.request.id":"9a24c6ca-0db5-4409-8892-977a5ddccc14","event.duration":45491,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-15T11:36:19.808Z","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":65},"message":"","service.name":"apm-server","source.address":"xxxxxx","grpc.request.method":"/opentelemetry.proto.collector.trace.v1.TraceService/Export","event.duration":153781,"grpc.response.status_code":"OK","ecs.version":"1.6.0"}

However only the events send using grpc show up in kibana, I suspect that I need to provide a endpoint other than root / in the url but I'm not sure what is has to be.

Knowing I had a working agent I went back to connecting to Elastic Cloud and ended up deleting the port from the url which fixed the connection issue.
So I can now actually see the trace information from the OpenTelemetry agent in kibana.

@GregKalapos thank you for the helpful tips.