.NET core app unsuccessful transfer to APM server on Kubernetes

[greenfield]

Kibana version: 7.11

Elasticsearch version: 7.11

APM Server version: 7.11

APM Agent language and version: .NET_Core/3.1.12

Original install method (e.g. download page, yum, deb, from source, etc.) and version: ECK operator via https://download.elastic.co/downloads/eck/1.4.0/all-in-one.yaml

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): Unable to send APM Agent data to APM server hosted on Kubernetes. The appsettings.json contains the public IP of the APM server. Should I be able to curl the public IP of the APM server? My config is using out-of-the-box self-signed certs.

Steps to reproduce:

1. Install APM server via ECK
2. Create .NET Core app & add nuget package
3. Update appsettings.json
4. Update startup.cs
5. Create container and deploy in seperate namespace

appsettings.json

{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information",
"Elastic.Apm": "Debug"
}
},
"AllowedHosts": "*",
"ElasticApm":
{
"ServerUrls": "http://20.73.125.50:8200",
"SecretToken": "jN5H8wSRla50297t22vtfto0",
"ServiceName": "myapp1"
}
}

Startup.cs
using Elastic.Apm.NetCoreAll;
...
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseAllElasticApm(Configuration);
...

Kubernetes svc overview
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 443/TCP 77m
elastic-system apmserver-apm-http LoadBalancer 10.0.106.60 20.73.125.50 8200:31985/TCP 68m
elastic-system elastic-webhook-server ClusterIP 10.0.181.159 443/TCP 69m
elastic-system elasticsearch-es-default ClusterIP None 9200/TCP 68m
elastic-system elasticsearch-es-http LoadBalancer 10.0.38.17 20.73.124.228 9200:32553/TCP 68m
elastic-system elasticsearch-es-transport ClusterIP None 9300/TCP 68m
elastic-system kibana-kb-http LoadBalancer 10.0.92.26 20.73.125.44 5601:31085/TCP 68m
kube-system kube-dns ClusterIP 10.0.0.10 53/UDP,53/TCP 77m
kube-system metrics-server ClusterIP 10.0.112.50 443/TCP 77m
myapp1 myapp1 ClusterIP 10.0.13.193 80/TCP 65m

Verify access to APM public IP
curl -i https://20.73.125.50:8200
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

.NET Core container logs
fail: Elastic.Apm[0]
{CentralConfigFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: <null>'. URL: http://20.73.125.50:8200/config/v1/agents?service.name=myapp1&service.environment=Production'. Apm Server base URL: `http://20.73.125.50:8200/'. WaitInterval: 5m. dbgIterationsCount: 1.
+-> Request:
Method: GET, RequestUri: 'http://20.73.125.50:8200/config/v1/agents?service.name=myapp1&service.environment=Production', Version: 1.1, Content: , Headers:
{
User-Agent: elasticapm-dotnet/1.7.1
User-Agent: System.Net.Http/4.700.21.6905
User-Agent: .NET_Core/3.1.12
Authorization: Bearer jN5H8wSRla50297t22vtfto0
}
+-> Response:
StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.0, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
}
+-> Response body [length: 48]:
Client sent an HTTP request to an HTTPS server.

warn: Elastic.Apm[0]
{PayloadSenderV2} Failed reading APM server info, response from server: BadRequest
dbug: Elastic.Apm[0]
{PayloadSenderV2} Enqueued MetricSet. newEventQueueCount: 1. MaxQueueEventCount: 1000. MetricSet: Elastic.Apm.Metrics.MetricSet.
dbug: Elastic.Apm[0]
{MetricsCollector} Metrics collected: MetricSample{system.process.cpu.total.norm.pct: 0.017932134045491432}, MetricSample{system.cpu.total.norm.pct: 0.15930999492643327}, MetricSample{system.process.memory.size: 9618743296}, MetricSample{system.process.memory.rss.bytes: 91856896}, MetricSample{system.memory.total: 8349163520}, MetricSample{system.memory.actual.free: 5830754304}, MetricSample{system.process.cgroup.memory.stats.inactive_file.bytes: 0}, MetricSample{system.process.cgroup.memory.mem.usage.bytes: 35168256}, MetricSample{system.process.cgroup.memory.mem.limit.bytes: 2147483648}
warn: Elastic.Apm[0]
{PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (http://20.73.125.50:8200/):
Elastic.Apm.Metrics.MetricSet

[greenfield]

I added the following to the eck apm server config:

http:

    tls:

      selfSignedCertificate:

        disabled: true

This solved the issue. Metrics are now forwarded to the APM server. BUT I don't see anything in the Kibana APM portal. Services and Traces remain empty.

Below the log output of the .NET Core application:

dbug: Elastic.Apm[0]
{CentralConfigFetcher} Combined absolute URL for APM Server get central configuration endpoint: `http://20.76.20.108:8200/config/v1/agents?service.name=myapp1&service.environment=Production'. Service: Service{Name: myapp1, Version: 1.0.0, Environment: Production, Runtime: Runtime{Name: .NET Core, Version: 3.1.12}, Framework: null, Agent: AgentC{Name: dotnet, Version: 1.7.1}, Language: null, Node: Node{ConfiguredName: null}}.
dbug: Elastic.Apm[0]

dbug: Elastic.Apm[0]
{MetricsCollector} Metrics collected: MetricSample{system.process.cpu.total.norm.pct: 0.0006667152524294954}, MetricSample{system.cpu.total.norm.pct: 0.1652765957446809}, MetricSample{system.process.memory.size: 9635901440}, MetricSample{system.process.memory.rss.bytes: 111722496}, MetricSample{system.memory.total: 8349163520}, MetricSample{system.memory.actual.free: 5869801472}, MetricSample{system.process.cgroup.memory.stats.inactive_file.bytes: 0}, MetricSample{system.process.cgroup.memory.mem.usage.bytes: 54075392}, MetricSample{system.process.cgroup.memory.mem.limit.bytes: 2147483648}
dbug: Elastic.Apm[0]
{PayloadSenderV2} Sent items to server:
Elastic.Apm.Metrics.MetricSet
dbug: Elastic.Apm[0]
{PayloadSenderV2} Enqueued MetricSet. newEventQueueCount: 1. MaxQueueEventCount: 1000. MetricSet: Elastic.Apm.Metrics.MetricSet.
dbug: Elastic.Apm[0]
{MetricsCollector} Metrics collected: MetricSample{system.process.cpu.total.norm.pct: 0.0009999753472744385}, MetricSample{system.cpu.total.norm.pct: 0.17254835425856807}, MetricSample{system.process.memory.size: 9635901440}, MetricSample{system.process.memory.rss.bytes: 111722496}, MetricSample{system.memory.total: 8349163520}, MetricSample{system.memory.actual.free: 5870059520}, MetricSample{system.process.cgroup.memory.stats.inactive_file.bytes: 0}, MetricSample{system.process.cgroup.memory.mem.usage.bytes: 54255616}, MetricSample{system.process.cgroup.memory.mem.limit.bytes: 2147483648}
dbug: Elastic.Apm[0]
{PayloadSenderV2} Sent items to server:
Elastic.Apm.Metrics.MetricSet

[greenfield]

I paste below 2 queries via DEV TOOLS:
GET apm-*/_search
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 6,
"successful" : 6,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 34,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "apm-7.11.0-metric-000001",
"_type" : "_doc",
"_id" : "Y30fsXcBfz0eGlIAWTwC",
"_score" : 1.0,
"_source" : {
"container" : {
"id" : "006d970e58a069cf123d3d10287404716e81f4c9b57b53956dd9df26d0ef03fb"
},
"kubernetes" : {
"pod" : {
"uid" : "006d970e58a069cf123d3d10287404716e81f4c9b57b53956dd9df26d0ef03fb",
"name" : "theapp1-7c8c5db7f5-2cmkd"
}
},
"agent" : {
"name" : "dotnet",
"version" : "1.7.1"
},
"processor" : {
"name" : "metric",
"event" : "metric"
},
GET /_template/apm-7.11.0
{
"apm-7.11.0" : {
"order" : 1,
"index_patterns" : [
"apm-7.11.0*"
],
"settings" : {
"index" : {
"codec" : "best_compression",
"mapping" : {
"total_fields" : {
"limit" : "2000"
}
},
"refresh_interval" : "5s",
"number_of_shards" : "1",
"max_docvalue_fields_search" : "200",
"query" : {
"default_field" : [
"message",
"tags",
"agent.ephemeral_id",
"agent.id",
"agent.name",
"agent.type",
"agent.version",

logs APM server container:
41932-10ec-49ed-9931-61ef22e2de55","event.duration":104900,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-02-17T18:12:35.533Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.18","source.address":"11.0.1.4","http.request.body.bytes":0,"http.request.id":"263e62c4-afb4-4642-88f1-99254d727176","event.duration":110801,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-02-17T18:12:36.788Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-dotnet/1.7.1 System.Net.Http/4.700.21.6905 .NET_Core/3.1.12","source.address":"11.0.1.4","http.request.body.bytes":1509,"http.request.id":"be2c5663-fe3a-47bc-85dd-17cb71d1b910","event.duration":219901,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-02-17T18:12:38.856Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"not modified","url.original":"/config/v1/agents?service.name=myapp1&service.environment=Production","http.request.method":"GET","user_agent.original":"elasticapm-dotnet/1.7.1 System.Net.Http/4.700.21.6905 .NET_Core/3.1.12","source.address":"11.0.1.4","http.request.body.bytes":0,"http.request.id":"783f76ce-a2a6-4c3c-b5d0-469ce988fc06","event.duration":88138289,"http.response.status_code":304,"ecs.version":"1.6.0"}

[GregKalapos]

Hi,

regarding the failing connection to APM Server:
supporting self signed certificates is already work in progress for the .NET Agent: Support self-signed certificates · Issue #1171 · elastic/apm-agent-dotnet · GitHub

regarding not seeing transactions: based on the logs there isn't really any load on the app - if it's a web app, you need some requests, if it's not a web app, you need to manually start transactions.