Couldn't verify APM server status

[KatiukOleg (Oleg)]

Hi elatic team,
I'm trying to use APM with such releases:

Kibana version: 7.2.0

Elasticsearch version: 7.2.0

APM Server version: 7.2.0

APM Agent language and version: .Net APM Agent

And have some problems, I couldn't push logs to APM server and firstly want to verify APM server status.

I configured APM as was mentioned in documentation and instructions in Kibana page, so I install APM server with RPM instructions and edit next fields only ->

output.elasticsearch:
hosts: ["localhost:9200"]
username: myusername
password: mypassword

I used localhost, cause APM server is located inside the same pod as Elasticsearch. Pod exist in Azure Kubernetes Service cluster.
Then I check APM-server via button ->

And got successful message.

According to official documentation (https://www.elastic.co/guide/en/apm/server/7.4/running-with-systemd.html) we can verify status via 'systemctl status apm-server' command. After I ran it, got this message ->
[root@azure-elasticsearch-prod-0 elasticsearch]# systemctl status apm-server
Failed to get D-Bus connection: Operation not permitted

It looks like my APM server is not configured properly.

Any thoughts?
Thanks

[gil (Gil Raphaelli)]

Hi @KatiukOleg. systemd is not used by the official Elastic docker container images so, assuming you're using those, you'll need an alternate approach to checking health. To simulate systemd, you should check that the process is running and tail the logs. I'd suggested enabling monitoring as well.

It sounds like apm-server->elasticsearch is ok since the kibana check is passing.

I couldn't push logs to APM server and firstly want to verify APM server status.

Which agent are you using? Are you able to enable logging from the agent to help narrow down the issue?

[KatiukOleg (Oleg)]

Hi @gil currently I'm using Elastic.Apm.EntityFrameworkCore 1.1.1 and Elastic.Apm.AspNetCore 1.1.1.

Here is my code snippet from appsettings:

"ElasticApm": {
"LogLevel": "Error",
"SecretToken": "",
"ServerUrls": "http://MyUser:MyPassword@elasticIP:8200",
"ServiceName": "MyService",
"ServiceVersion": "1.0.0"
}

During running application I got next logs:

Elastic.Apm:Warning: {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (http://MyUser:MyPassword@elasticIP:8200/):
Elastic.Apm.Metrics.MetricSet

System.Net.Http.HttpRequestException: Connection refused
Elastic APM .NET Agent: [CRITICAL] Exception thrown by logging implementation. Log message: `Failed sending events. Following events were not transferred successfully to the server ({ApmServerUrl}):

In Azure I add inbound rule in Network Security Group to allow traffic to 8200 port.

I'll try to check apm-server service inside elastic pod.

[KatiukOleg (Oleg)]

I found out that apm-server service was disabled

using 'systemctl enable apm-server' help me to enable apm-server.

After this I tried again to push logs to apm-server, but got the same errors as previous.

[gil (Gil Raphaelli)]

This bit looks suspicious:

"ServerUrls": "http://MyUser:MyPassword@elasticIP:8200",

is the ElasticIP that of apm-server or of Elasticsearch itself? I would not expect basic authentication for apm-server unless you had some custom proxy in front. Can you test connectivity to apm-server from your application server, if you have curl try: curl -i http://elasticIP:8200.

[KatiukOleg (Oleg)]

@gil, thanks. Yes MyUser:MyPassword part in ServerUrl is redundant. Should be removed. I think I chose bad approach initially. I installed APM-server as RPM package inside elasticssearch pod and forgot to edit elasticsearch YAML file to open port (8200) for APM-server. I'm going to set up APM-server on dedicated pod via deployment and service objects. Then return with a result.

[KatiukOleg (Oleg)]

Hi @gil, I tried to run APM as separated service in Azure Kubernetes Service with next deployment configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-apm-prod
  namespace: logging-system
  labels:
    env: production
    space: logging-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-apm-prod
  template:
    metadata:
      labels:
        app: azure-apm-prod
    spec:
      containers:
      - name: azure-apm-prod
        image: docker.elastic.co/apm/apm-server:7.4.0
        resources:
          limits:
            memory: 512Mi
            cpu: 500m
          requests:
            memory: 256Mi
            cpu: 200m
        ports:
        - containerPort: 8200
        env:
        - name: OUTPUT_ELASTICSEARCH_HOSTS
          value: http://azure-elasticsearch-prod:9200
        - name: OUTPUT_ELASTICSEARCH_USERNAME
          value: "elasticUser"
        - name: OUTPUT_ELASTICSEARCH_PASSWORD
          value: "elasticsPassword"
        - name: APM_SERVER_RUM_ENABLED
          value: "true"

But unfortunately I couldnt send items to APM server. Got next errors:

Exception thrown: 'Elastic.Apm.BackendComm.CentralConfigFetcher.FailedToFetchConfigException' in Elastic.Apm.dll
Exception thrown: 'Elastic.Apm.BackendComm.CentralConfigFetcher.FailedToFetchConfigException' in Elastic.Apm.dll
e[40me[37mdbuge[39me[22me[49m: Elastic.Apm[0]
      {CentralConfigFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `<null>'. URL: `http://52.157.137.240:8200/config/v1/agents?service.name=MissionControl&service.environment=Development'. Apm Server base URL: `http://52.157.137.240:8200/'. WaitInterval: 5m. dbgIterationsCount: 2.
      +-> Request:
          Method: GET, RequestUri: 'http://APMurl:8200/config/v1/agents?service.name=ServiceName&service.environment=Development', Version: 2.0, Content: <null>, Headers:
          {
            User-Agent: elasticapm-dotnet/1.1.1
            User-Agent: System.Net.Http/4.6.28008.03
            User-Agent: .NET_Core/2.2.7
            Request-Id: |24f69f431f2ffc44b99c9276ccb8ef76.8b178e7b_
          }
      +-> Response:
          StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
          {
            X-Content-Type-Options: nosniff
            Date: Fri, 18 Oct 2019 12:47:51 GMT
            Content-Type: text/plain; charset=utf-8
            Content-Length: 52
          }
      +-> Response body [length: 52]:
          {"error":"forbidden request: endpoint is disabled"}
      
Elastic.Apm.BackendComm.CentralConfigFetcher+FailedToFetchConfigException: HTTP status code is Forbidden which most likely means that APM Server supports the central configuration endpoint but Kibana connection is not enabled
   at Elastic.Apm.BackendComm.CentralConfigFetcher.InterpretResponseStatusCode(HttpResponseMessage httpResponse)
   at Elastic.Apm.BackendComm.CentralConfigFetcher.ProcessHttpResponse(HttpResponseMessage httpResponse, String httpResponseBody)
   at Elastic.Apm.BackendComm.CentralConfigFetcher.WorkLoopIteration()

It looks like some configuration is missed ....
Any thoughts?

Best Regards,
Oleg

[gil (Gil Raphaelli)]

Hey @KatiukOleg, sorry for the delay. It looks like central configuration is not enabled, to that you should configure a kibana host. I wouldn't expect that to prevent ingestion of trace events though, do you see any data in elasticsearch?

[KatiukOleg (Oleg)]

Hi @gil, after several times I configured APM server and now can push data to elasticsearch.
I figured out that environment variables in deployment.yaml file didn't work for me (container logs in Azure were rather informative) and apm-server.yml file on host is existed in readonly mode. I couldn't edit it either.

So I looked at Narendiran D post (https://medium.com/logistimo-engineering-blog/how-did-i-use-apm-in-kubernetes-ecosystem-8f22d52beb03) and configured ConfigMap object (all environment variables were migrated to ConfigMap).

After these manipulations everything works fine.

[gil (Gil Raphaelli)]

Glad to hear it, thanks for following up.