{PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server

Elastic Observability APM
1

Kibana version: None

Elasticsearch version: None

APM Server version: 7.10.1 (amd64), libbeat 7.10.1 [b7c209e80c4674603447458e62963ed5246b5297 built 2020-12-04 22:07:34 +0000 UTC]

APM Agent language and version: .Net Core with Elastic.Apm.NetCoreAll from 1.7.0 to 1.9.0 (upgraded because it also wasn't working with 1.7.0)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: apt install apm-server

Fresh install or upgraded from other version?: Fresh install

Is there anything special in your setup? Logstash output, changed agent configuration (disabled kibana and elasticsearch).

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Configured a VM on Azure with Ubuntu 18.04 with Logstash and APM Server to receive events from an APM Agent for a .Net Core API hosted in Azure App Services.

APM Server config:

apm-server:
  host: "0.0.0.0:8200"

  ilm:
    enabled: "false"

output.logstash:
  enabled: true
  hosts: ["localhost:5044"]
  escape_html: true

.Net Core appsettings:

"ElasticApm": {
    "ServerUrl": "https://AzureVmUrl",
    "ServiceName": "XemanticA.Web.Backend"
  }

.Net Core packages for logs:

Elastic.Apm.NetCoreAll 1.7.0 -> 1.9.0
NLog.Web.AspNetCore 4.9.3

TCP Connection to the Azure VM through port 8200 is enabled and working.

When running the app, the Elastic package or APM Agent can't send events to the APM Server.

Provide logs and/or server output (if relevant):
Changed APM Server url to: AzureVmUrl
Changed app name to: WebApp
Changed SQL Queries to: SQLQuery

Full log (pastebin)

2

Hi @Rolfen,

based on the attached logs, to me this looks that the .NET application with the agent can't communicate with the APM Server. It already fails when it tries to read APM server info:

[2021-04-06 12:21:23.7629] - [WARN] - [Elastic.Apm.Extensions.Hosting.NetCoreLogger.Log (0)] - {PayloadSenderV2} Failed reading APM server info

And then when the agent tries to send the events, it fails as well:

[2021-04-06 12:21:45.2071] - [WARN] - [Elastic.Apm.Extensions.Hosting.NetCoreLogger.Log (0)] - {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (https://AzureVmUrl/):

For both of these logs, the agent also prints the exception which happens during the HTTP communication - I don't see those in your logs. How do you collect these logs? Can you print the exceptions? That would reveal more.

Bot overall this looks to me that the agent can't talk to the APM server and seems like an HTTP connection issue - I'd expect the exception being something about the connection.

3

I'm getting the logs from the output console of Visual Studio Enterprise, Its not printing the exceptions and thats why I'm lost, because I haven't been able to narrow down the problem. The Elastic log level is set to Debug, shouldn't it be enough to capture exceptions if they are thrown?

Also, as you are specifying that the communications is through http/https I checked the connection to the server and it's working fine for ports 80 and 442, but still can't communicate with the APM Server.

4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

5

Based on the APM server config, APM server is listening on port 8200; Does "https://AzureVmUrl" in the APM .Net agent ServerUrl also contain port 8200?

7

Before I had the ServerUrl like "https://AzureVmUrl:8200" and it already wasn't working. Somewhere I read that adding the ":8200" to the ServerUrl didn't matter so I got rid of it, but now I tried again adding the port, and still, it can't connect.

8

What's the error in the agent logs?

Are you able to make a GET request to https://AzureVmUrl:8200 from the VM and get a successful response?