APM agent not communicating with APM-Server

Elastic Observability APM
,
1

APM agent cannot contact APM-Server. The servers with the APM agent hard coded in web.config (in .NET FullFramework) have been giving us the following error message in Event Viewer:

Timestamp: 11/30/2020 2:41:58 PM
Message: {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (http://localhost:8200/):
Elastic.Apm.Metrics.MetricSet

The following is a snippet of the web.config:

ApmWebConfig
ApmWebConfig701×64 5.39 KB

Have been troubleshooting for communication between the APM-Server and the APM agent, but logs aren't giving us details. Was hoping someone had any ideas.

Thank you

2

Hi @GeoffRx,

the error message says http://localhost:8200/, but your web.config shows a ServerUrls value with https - that's strange. I suspect the removed part of the value isn't localhost, so it seems the agent does not pick up the value.

Do you maybe see an error during startup regarding parsing the ServerUrls value? Or do you maybe have the same value configured as an environment variable as well?

3

Hey @GregKalapos,

Thanks for the timely response. We did not see any logs regarding parsing the ServerUrls value. Fortunately, however, we have finally gotten the APM agent communicating with the APM Server. We had do disable SSL for now, just to see if it would indeed send data.

So we believe there is an issue with certificates and authentication. We generated the certs with the provided elasticsearch-certutil, but we're thinking we may have to obtain another cert that isn't self-signed. Another oddity is even though APM is now receiving data, we are still getting the same error message posted in my first post.

A bit of a dilemma but when we find the answer, I will update this post to share with others. Thanks again Greg,

Geoff

4

Thanks for the update Geoff

This sounds very strange - definitely not normal behaviour. The agent by default tries to connect to localhost:8200, but if you set another ServerUrls value, then it should not do this. Let us know if you can find out something. If this persists and it's reproducible, then this sounds like a bug, so we'll look into it if there is more info.

5

@GregKalapos,

Wanted to update you on this issue. We re-enabled SSL and commented out the secret token in the apm-server.yml. We are receiving data from the .NET APM Agent to the APM Server but are still receiving the following error message in Windows Event Viewer on all servers with the APM agent installed:

Message: {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (http://localhost:8200/):

However, the agent is sending the data we need and SSL is enabled, so, we are satisfied with the outcome. Just wanted to let you know in case there is indeed a bug or need for further investigation.

6

I am seeing the same thing with Windows 10 endpoint now following the installation guide.

PS C:\Program Files\Elastic-Agent> .\elastic-agent enroll https://192.168.111.224:443 NnI3bS1uVUJWWWZFUlhqWjFmRDY6WnZDZ0ZhaUVUZkdjVVA4ajVyZTc4dw==
The Elastic Agent is currently in BETA and should not be used in production

2020-12-02T11:58:07.966-0600 DEBUG kibana/client.go:170 Request method: POST, path: /api/fleet/agents/enroll
Error: fail to enroll: fail to execute request to Kibana: Post "https://192.168.111.224:443/api/fleet/agents/enroll?": x509: certificate signed by unknown authority
PS C:\Program Files\Elastic-Agent>

As you can see when trying to enroll the agent we are getting a x509 cert error. If I take out the HTTPS it will not throw this error

PS C:\Program Files\Elastic-Agent> .\elastic-agent enroll http://192.168.111.224:443 NnI3bS1uVUJWWWZFUlhqWjFmRDY6WnZDZ0ZhaUVUZkdjVVA4ajVyZTc4dw==
The Elastic Agent is currently in BETA and should not be used in production

Error: Error: connection to Kibana is insecure, strongly recommended to use a secure connection (override with --insecure)
PS C:\Program Files\Elastic-Agent>

and we get this error when trying to register as a service.

.\install-service-elastic-agent.ps1 : The term '.\install-service-elastic-agent.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
At line:1 char:1

  • .\install-service-elastic-agent.ps1
  •   + CategoryInfo          : ObjectNotFound: (.\install-service-elastic-agent.ps1:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException
7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.