Troubleshooting APM Agent to Fleet Server

Elastic Observability APM
1

Hi,
I'm trying to set up Elastic APM on a dotnet mvc app. Testing with a dotnet core console app but my intended setup is for a regular dotnet framework project in mvc.

I have Elasticsearch and Kibana set up.
I have my Elastic Agent running Fleet set up and self-signing ssl certs.
Fleet status is healthy. Diagnostics look clean.

However, when I go to run the sample apm-agent-dotnet console app I get

      {CentralConfigurationFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `<null>'. URL: `https://<sb-devora-apm-server--url>:8220/config/v1/agents?service.name=SampleConsoleNetCoreApp&service.environment=Production'. Apm Server base URL: `https://<sb-devora-apm-server--url>:8220/'. WaitInterval: 5m. dbgIterationsCount: 9.
      +-> Request:
          Method: GET, RequestUri: 'https://<sb-devora-apm-server--url>:8220/config/v1/agents?service.name=SampleConsoleNetCoreApp&service.environment=Production', Version: 1.1, Content: <null>, Headers:
          {
            User-Agent: apm-agent-dotnet/1.18.0
            User-Agent: (SampleConsoleNetCoreApp 1.0.0)
            User-Agent: System.Net.Http/4.700.22.47602
            User-Agent: .NET_Core/3.1.30
            Request-Id: |47ae3bf2-4d058de53621017d.
          }
      +-> Response:
          StatusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
          {
            X-Content-Type-Options: nosniff
            Date: Thu, 03 Nov 2022 19:22:48 GMT
            Content-Type: text/plain; charset=utf-8
            Content-Length: 19
          }
      +-> Response body [length: 19]:
          404 page not found

Also when I go to visit the url directly I can see the ssl cert in the browser but also receiving the 404 error.

Is this a config issue with the elastic server?

As far as I can tell my port is open on 8220. I'm not sure what a good next step to take would be. I see nothing on the receiving Elastic Agent / Fleet side so the payload is just not making it over. Meaning no errors to show on the Elastic Agent / Fleet side.

Another line in the debug output for my console app

} Failed sending event. Events intake API absolute URL: https://<sb-devora-apm-server--url>:8220/intake/v2/events. APM Server response: status code: NotFound, content: 
404 page not found
2

Hi Daniel,

did you use NuGet to deploy the APM .NET Agent to your application?
Can you also check your configuration for me, please?
At the very minimum you need to configure ServerUrl and SecretToken (see documentation).

-Wolfgang

3

Thank you for the fast response.

I did not use NuGet for the sample application I'm testing. But I did use it for my main app.

Sample app

{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Information"
    }
  },

  "ElasticApm": {
    "ServerUrls": "https://<sb-devora-fleet--url>:8220",
    "SecretToken": "elk==",
    "VerifyServerCert": false
  },

  "Console": {
    "IncludeScopes": true // Required to use Scopes.
  }
}

My app

	<add key="ElasticApm:ServerUrl" value="https://<sb-devora-fleet--url>:8220" />
	<add key="ElasticApm:SecretToken" value="elk==" />
4

Hi Daniel,
the 404 error suggests a problem with your setup.

If you navigate to Integrations| APM

image
image3138×766 127 KB

Are you using the correct settings from there?

image
image1920×1627 135 KB

5

Perhaps getting closer?

  • Config without VerifyServerCert
  "ElasticApm": {
    "ServerUrls": "https://xx.xx.xx.xx:8220",
    "SecretToken": "elkxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==",
    "ServiceName": "SAMPLECONSOLENETCORE",
    "Environment": "production"
  },

Provides exception

System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
  • Config with VerifyServerCert
  "ElasticApm": {
    "ServerUrls": "https://xx.xx.xx.xx:8220",
    "SecretToken": "elkxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==",
    "ServiceName": "SAMPLECONSOLENETCORE",
    "Environment": "production",
    "VerifyServerCert": false
  },

Provides exception

      {CentralConfigurationFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `<null>'. URL: `https://xx.xx.xx.xx:8220/config/v1/agents?service.name=SAMPLECONSOLENETCORE&service.environment=production'. Apm Server base URL: `https://10.0.2.23:8220/'. WaitInterval: 5m. dbgIterationsCount: 1.
      +-> Request:
          Method: GET, RequestUri: 'https://xx.xx.xx.xx:8220/config/v1/agents?service.name=SAMPLECONSOLENETCORE&service.environment=production', Version: 1.1, Content: <null>, Headers:
          {
            User-Agent: apm-agent-dotnet/1.18.0
            User-Agent: (SAMPLECONSOLENETCORE 1.0.0)
            User-Agent: System.Net.Http/4.700.22.47602
            User-Agent: .NET_Core/3.1.30
            Authorization: [REDACTED]
          }
      +-> Response:
          StatusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
          {
            X-Content-Type-Options: nosniff
            Date: Fri, 04 Nov 2022 15:07:38 GMT
            Content-Type: text/plain; charset=utf-8
            Content-Length: 19
          }
      +-> Response body [length: 19]:
          404 page not found

Elastic.Apm.BackendComm.CentralConfig.CentralConfigurationFetcher+FailedToFetchConfigException: HTTP status code is Not Found (404) which most likely means that APM Server is an old (pre 7.3) version which doesn't support the central configuration endpoint
   at Elastic.Apm.BackendComm.CentralConfig.CentralConfigurationResponseParser.InterpretResponseStatusCode(HttpResponseMessage httpResponse, WaitInfoS waitInfo) in C:\Users\devora\source\pocs\apm-agent-dotnet-main\apm-agent-dotnet-main\src\Elastic.Apm\BackendComm\CentralConfig\CentralConfigurationResponseParser.cs:line 135
   at Elastic.Apm.BackendComm.CentralConfig.CentralConfigurationResponseParser.ParseHttpResponse(HttpResponseMessage httpResponse, String httpResponseBody) in C:\Users\devora\source\pocs\apm-agent-dotnet-main\apm-agent-dotnet-main\src\Elastic.Apm\BackendComm\CentralConfig\CentralConfigurationResponseParser.cs:line 39

Should I be able to hit the request urls in a browser as a ssl verification check?

I'm using the self-signed certificate option. When I do visit within the browser I do receive the same 404 type error

6

Hi Daniel,

can you try a tool like curl or wget to hit the URL configured at ElasticApm.ServerUrls?
What's the response you get there?

If you have problems with your self-signed certificate, these tools have options like (--insecure, or --no-check-certificate) that you can try.

If this does not work, there is no change for the .NET APM Agent either to report data to this endpoint.

7 
$ curl  https://xx.xx.xx.xx:8220 --insecure
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    19  100    19    0     0    236      0 --:--:-- --:--:-- --:--:--   292404 page not found

Same 404 issue with curl.

Did I set up my Fleet Server + Elastic Agent incorrectly?

Fleet screen in Kibana shows healthy.
Elastic-Agent status shows healthy.

Elastic + Kibana + Elastic Agent all v. 8.3.2

Fleet v 1.2.0

8

An update. I switched over to the setup for the legacy APM server and was successful on that approach. I can go with this for now but eventually it seems we will have to migrate over to the Fleet + Agent approach.

9

It looks like you are using an incorrect URL (at least an incorrect port number) for your APM server URL configuration.
APM server typically uses port 8200 (while here 8220 is used).

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.