{PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server

Kibana version: None

Elasticsearch version: None

APM Server version: 7.10.1 (amd64), libbeat 7.10.1 [b7c209e80c4674603447458e62963ed5246b5297 built 2020-12-04 22:07:34 +0000 UTC]

APM Agent language and version: .Net Core with Elastic.Apm.NetCoreAll from 1.7.0 to 1.9.0 (upgraded because it also wasn't working with 1.7.0)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: apt install apm-server

Fresh install or upgraded from other version?: Fresh install

Is there anything special in your setup? Logstash output, changed agent configuration (disabled kibana and elasticsearch).

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Configured a VM on Azure with Ubuntu 18.04 with Logstash and APM Server to receive events from an APM Agent for a .Net Core API hosted in Azure App Services.

APM Server config:

apm-server:
  host: "0.0.0.0:8200"

  ilm:
    enabled: "false"

output.logstash:
  enabled: true
  hosts: ["localhost:5044"]
  escape_html: true

.Net Core appsettings:

"ElasticApm": {
    "ServerUrl": "https://AzureVmUrl",
    "ServiceName": "XemanticA.Web.Backend"
  }

.Net Core packages for logs:

Elastic.Apm.NetCoreAll 1.7.0 -> 1.9.0
NLog.Web.AspNetCore 4.9.3

TCP Connection to the Azure VM through port 8200 is enabled and working.

When running the app, the Elastic package or APM Agent can't send events to the APM Server.

Provide logs and/or server output (if relevant):
Changed APM Server url to: AzureVmUrl
Changed app name to: WebApp
Changed SQL Queries to: SQLQuery

Full log (pastebin)

Hi @Rolfen,

based on the attached logs, to me this looks that the .NET application with the agent can't communicate with the APM Server. It already fails when it tries to read APM server info:

[2021-04-06 12:21:23.7629] - [WARN] - [Elastic.Apm.Extensions.Hosting.NetCoreLogger.Log (0)] - {PayloadSenderV2} Failed reading APM server info

And then when the agent tries to send the events, it fails as well:

[2021-04-06 12:21:45.2071] - [WARN] - [Elastic.Apm.Extensions.Hosting.NetCoreLogger.Log (0)] - {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (https://AzureVmUrl/):

For both of these logs, the agent also prints the exception which happens during the HTTP communication - I don't see those in your logs. How do you collect these logs? Can you print the exceptions? That would reveal more.

Bot overall this looks to me that the agent can't talk to the APM server and seems like an HTTP connection issue - I'd expect the exception being something about the connection.

I'm getting the logs from the output console of Visual Studio Enterprise, Its not printing the exceptions and thats why I'm lost, because I haven't been able to narrow down the problem. The Elastic log level is set to Debug, shouldn't it be enough to capture exceptions if they are thrown?

Also, as you are specifying that the communications is through http/https I checked the connection to the server and it's working fine for ports 80 and 442, but still can't communicate with the APM Server.