Kibana version: 7.11
Elasticsearch version: 7.11
APM Server version: 7.11
APM Agent language and version: .NET_Core/3.1.12
Original install method (e.g. download page, yum, deb, from source, etc.) and version: ECK operator via https://download.elastic.co/downloads/eck/1.4.0/all-in-one.yaml
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): Unable to send APM Agent data to APM server hosted on Kubernetes. The appsettings.json contains the public IP of the APM server. Should I be able to curl the public IP of the APM server? My config is using out-of-the-box self-signed certs.
Steps to reproduce:
- Install APM server via ECK
- Create .NET Core app & add nuget package
- Update appsettings.json
- Update startup.cs
- Create container and deploy in seperate namespace
appsettings.json
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information",
"Elastic.Apm": "Debug"
}
},
"AllowedHosts": "*",
"ElasticApm":
{
"ServerUrls": "http://20.73.125.50:8200",
"SecretToken": "jN5H8wSRla50297t22vtfto0",
"ServiceName": "myapp1"
}
}
Startup.cs
using Elastic.Apm.NetCoreAll;
...
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseAllElasticApm(Configuration);
...
Kubernetes svc overview
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 443/TCP 77m
elastic-system apmserver-apm-http LoadBalancer 10.0.106.60 20.73.125.50 8200:31985/TCP 68m
elastic-system elastic-webhook-server ClusterIP 10.0.181.159 443/TCP 69m
elastic-system elasticsearch-es-default ClusterIP None 9200/TCP 68m
elastic-system elasticsearch-es-http LoadBalancer 10.0.38.17 20.73.124.228 9200:32553/TCP 68m
elastic-system elasticsearch-es-transport ClusterIP None 9300/TCP 68m
elastic-system kibana-kb-http LoadBalancer 10.0.92.26 20.73.125.44 5601:31085/TCP 68m
kube-system kube-dns ClusterIP 10.0.0.10 53/UDP,53/TCP 77m
kube-system metrics-server ClusterIP 10.0.112.50 443/TCP 77m
myapp1 myapp1 ClusterIP 10.0.13.193 80/TCP 65m
Verify access to APM public IP
curl -i https://20.73.125.50:8200
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
.NET Core container logs
fail: Elastic.Apm[0]
{CentralConfigFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: <null>'. URL:
http://20.73.125.50:8200/config/v1/agents?service.name=myapp1&service.environment=Production'. Apm Server base URL: `http://20.73.125.50:8200/'. WaitInterval: 5m. dbgIterationsCount: 1.
+-> Request:
Method: GET, RequestUri: 'http://20.73.125.50:8200/config/v1/agents?service.name=myapp1&service.environment=Production', Version: 1.1, Content: , Headers:
{
User-Agent: elasticapm-dotnet/1.7.1
User-Agent: System.Net.Http/4.700.21.6905
User-Agent: .NET_Core/3.1.12
Authorization: Bearer jN5H8wSRla50297t22vtfto0
}
+-> Response:
StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.0, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
}
+-> Response body [length: 48]:
Client sent an HTTP request to an HTTPS server.
warn: Elastic.Apm[0]
{PayloadSenderV2} Failed reading APM server info, response from server: BadRequest
dbug: Elastic.Apm[0]
{PayloadSenderV2} Enqueued MetricSet. newEventQueueCount: 1. MaxQueueEventCount: 1000. MetricSet: Elastic.Apm.Metrics.MetricSet.
dbug: Elastic.Apm[0]
{MetricsCollector} Metrics collected: MetricSample{system.process.cpu.total.norm.pct: 0.017932134045491432}, MetricSample{system.cpu.total.norm.pct: 0.15930999492643327}, MetricSample{system.process.memory.size: 9618743296}, MetricSample{system.process.memory.rss.bytes: 91856896}, MetricSample{system.memory.total: 8349163520}, MetricSample{system.memory.actual.free: 5830754304}, MetricSample{system.process.cgroup.memory.stats.inactive_file.bytes: 0}, MetricSample{system.process.cgroup.memory.mem.usage.bytes: 35168256}, MetricSample{system.process.cgroup.memory.mem.limit.bytes: 2147483648}
warn: Elastic.Apm[0]
{PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (http://20.73.125.50:8200/):
Elastic.Apm.Metrics.MetricSet