# Couldn't verify APM server status

**URL:** <https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511>\
**Category:** APM\
**Tags:** server\
**Created:** [October 7, 2019, 9:49am UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511 "2019-10-07T09:49:58Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![KatiukOleg](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@KatiukOleg](https://discuss.elastic.co/u/KatiukOleg)\
**Post date:** [October 7, 2019, 9:49am UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/1 "2019-10-07T09:49:58Z")

</div>

Hi elatic team,  
I'm trying to use APM with such releases:

**Kibana version** : 7.2.0

**Elasticsearch version** : 7.2.0

**APM Server version** : 7.2.0

**APM Agent language and version** : .Net APM Agent

And have some problems, I couldn't push logs to APM server and firstly want to verify APM server status.

I configured APM as was mentioned in documentation and instructions in Kibana page, so I install APM server with RPM instructions and edit next fields only -\>

output.elasticsearch:  
hosts: ["localhost:9200"]  
username: myusername  
password: mypassword

I used localhost, cause APM server is located inside the same pod as Elasticsearch. Pod exist in Azure Kubernetes Service cluster.  
Then I check APM-server via button -\>

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/e/be2dfefad0ada3ea0d990e45960c90e2d23ce47e.png)

And got successful message.

According to official documentation ([https://www.elastic.co/guide/en/apm/server/7.4/running-with-systemd.html](https://www.elastic.co/guide/en/apm/server/7.4/running-with-systemd.html)) we can verify status via 'systemctl status apm-server' command. After I ran it, got this message -\>  
[root@azure-elasticsearch-prod-0 elasticsearch]# systemctl status apm-server  
Failed to get D-Bus connection: Operation not permitted

It looks like my APM server is not configured properly.

Any thoughts?  
Thanks

---

<div class="post-metadata">

**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\
**Post date:** [October 7, 2019, 3:51pm UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/2 "2019-10-07T15:51:59Z")

</div>

Hi @KatiukOleg. systemd is not used by the official Elastic docker container images so, assuming you're using those, you'll need an alternate approach to checking health. To simulate systemd, you should check that the process is running and tail the logs. I'd suggested enabling [monitoring](https://www.elastic.co/guide/en/apm/server/current/monitoring.html) as well.

It sounds like apm-server-\>elasticsearch is ok since the kibana check is passing.

> [@KatiukOleg](#):
>
> I couldn't push logs to APM server and firstly want to verify APM server status.

Which agent are you using? Are you able to enable logging from the agent to help narrow down the issue?

---

<div class="post-metadata">

**Author:** ![KatiukOleg](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@KatiukOleg](https://discuss.elastic.co/u/KatiukOleg)\
**Post date:** [October 8, 2019, 11:07am UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/3 "2019-10-08T11:07:13Z")

</div>

Hi @gil currently I'm using Elastic.Apm.EntityFrameworkCore 1.1.1 and Elastic.Apm.AspNetCore 1.1.1.

Here is my code snippet from appsettings:

> "ElasticApm": {  
> "LogLevel": "Error",  
> "SecretToken": "",  
> "ServerUrls": "[http://MyUser:MyPassword@elasticIP:8200](http://MyUser:MyPassword@elasticIP:8200)",  
> "ServiceName": "MyService",  
> "ServiceVersion": "1.0.0"  
> }

During running application I got next logs:

> Elastic.Apm:Warning: {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server ([http://MyUser:MyPassword@elasticIP:8200/](http://MyUser:MyPassword@elasticIP:8200/)):  
> Elastic.Apm.Metrics.MetricSet

> System.Net.Http.HttpRequestException: Connection refused  
> Elastic APM .NET Agent: [CRITICAL] Exception thrown by logging implementation. Log message: `Failed sending events. Following events were not transferred successfully to the server ({ApmServerUrl}):

In Azure I add inbound rule in Network Security Group to allow traffic to 8200 port.

I'll try to check apm-server service inside elastic pod.

---

<div class="post-metadata">

**Author:** ![KatiukOleg](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@KatiukOleg](https://discuss.elastic.co/u/KatiukOleg)\
**Post date:** [October 8, 2019, 12:27pm UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/4 "2019-10-08T12:27:11Z")

</div>

I found out that apm-server service was disabled

 ![APMserver_disabled](https://us1.discourse-cdn.com/elastic/original/3X/0/0/0077dd2030a7c24ff87f0597157453c8ba571717.png)

using 'systemctl enable apm-server' help me to enable apm-server.

 ![APMserver_enabled](https://us1.discourse-cdn.com/elastic/original/3X/9/1/916d8abd1f80d0503ef72503b5547a258cafaf8a.png)

After this I tried again to push logs to apm-server, but got the same errors as previous.

---

<div class="post-metadata">

**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\
**Post date:** [October 8, 2019, 6:03pm UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/5 "2019-10-08T18:03:53Z")

</div>

This bit looks suspicious:

> [@KatiukOleg](#):
>
> "ServerUrls": "[http://MyUser:MyPassword@elasticIP:8200](http://MyUser:MyPassword@elasticIP:8200)",

is the ElasticIP that of apm-server or of Elasticsearch itself? I would not expect basic authentication for apm-server unless you had some custom proxy in front. Can you test connectivity to apm-server from your application server, if you have `curl` try: `curl -i http://elasticIP:8200`.

---

<div class="post-metadata">

**Author:** ![KatiukOleg](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@KatiukOleg](https://discuss.elastic.co/u/KatiukOleg)\
**Post date:** [October 9, 2019, 9:50am UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/6 "2019-10-09T09:50:50Z")

</div>

@gil, thanks. Yes MyUser:MyPassword part in ServerUrl is redundant. Should be removed. I think I chose bad approach initially. I installed APM-server as RPM package inside elasticssearch pod and forgot to edit elasticsearch YAML file to open port (8200) for APM-server. I'm going to set up APM-server on dedicated pod via deployment and service objects. Then return with a result.

---

<div class="post-metadata">

**Author:** ![KatiukOleg](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@KatiukOleg](https://discuss.elastic.co/u/KatiukOleg)\
**Post date:** [October 18, 2019, 2:11pm UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/7 "2019-10-18T14:11:22Z")

</div>

Hi @gil, I tried to run APM as separated service in Azure Kubernetes Service with next deployment configuration

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-apm-prod
  namespace: logging-system
  labels:
    env: production
    space: logging-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-apm-prod
  template:
    metadata:
      labels:
        app: azure-apm-prod
    spec:
      containers:
      - name: azure-apm-prod
        image: docker.elastic.co/apm/apm-server:7.4.0
        resources:
          limits:
            memory: 512Mi
            cpu: 500m
          requests:
            memory: 256Mi
            cpu: 200m
        ports:
        - containerPort: 8200
        env:
        - name: OUTPUT_ELASTICSEARCH_HOSTS
          value: http://azure-elasticsearch-prod:9200
        - name: OUTPUT_ELASTICSEARCH_USERNAME
          value: "elasticUser"
        - name: OUTPUT_ELASTICSEARCH_PASSWORD
          value: "elasticsPassword"
        - name: APM_SERVER_RUM_ENABLED
          value: "true"

```

But unfortunately I couldnt send items to APM server. Got next errors:

```
Exception thrown: 'Elastic.Apm.BackendComm.CentralConfigFetcher.FailedToFetchConfigException' in Elastic.Apm.dll
Exception thrown: 'Elastic.Apm.BackendComm.CentralConfigFetcher.FailedToFetchConfigException' in Elastic.Apm.dll
e[40me[37mdbuge[39me[22me[49m: Elastic.Apm[0]
      {CentralConfigFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `<null>'. URL: `http://52.157.137.240:8200/config/v1/agents?service.name=MissionControl&service.environment=Development'. Apm Server base URL: `http://52.157.137.240:8200/'. WaitInterval: 5m. dbgIterationsCount: 2.
      +-> Request:
          Method: GET, RequestUri: 'http://APMurl:8200/config/v1/agents?service.name=ServiceName&service.environment=Development', Version: 2.0, Content: <null>, Headers:
          {
            User-Agent: elasticapm-dotnet/1.1.1
            User-Agent: System.Net.Http/4.6.28008.03
            User-Agent: .NET_Core/2.2.7
            Request-Id: |24f69f431f2ffc44b99c9276ccb8ef76.8b178e7b_
          }
      +-> Response:
          StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
          {
            X-Content-Type-Options: nosniff
            Date: Fri, 18 Oct 2019 12:47:51 GMT
            Content-Type: text/plain; charset=utf-8
            Content-Length: 52
          }
      +-> Response body [length: 52]:
          {"error":"forbidden request: endpoint is disabled"}
      
Elastic.Apm.BackendComm.CentralConfigFetcher+FailedToFetchConfigException: HTTP status code is Forbidden which most likely means that APM Server supports the central configuration endpoint but Kibana connection is not enabled
   at Elastic.Apm.BackendComm.CentralConfigFetcher.InterpretResponseStatusCode(HttpResponseMessage httpResponse)
   at Elastic.Apm.BackendComm.CentralConfigFetcher.ProcessHttpResponse(HttpResponseMessage httpResponse, String httpResponseBody)
   at Elastic.Apm.BackendComm.CentralConfigFetcher.WorkLoopIteration()

```

It looks like some configuration is missed ....  
Any thoughts?

Best Regards,  
Oleg

---

<div class="post-metadata">

**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\
**Post date:** [October 22, 2019, 2:52pm UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/8 "2019-10-22T14:52:58Z")

</div>

Hey @KatiukOleg, sorry for the delay. It looks like central configuration is not enabled, to that you should [configure a kibana host](https://www.elastic.co/guide/en/apm/server/current/setup-kibana-endpoint.html). I wouldn't expect that to prevent ingestion of trace events though, do you see any data in elasticsearch?

---

<div class="post-metadata">

**Author:** ![KatiukOleg](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@KatiukOleg](https://discuss.elastic.co/u/KatiukOleg)\
**Post date:** [October 25, 2019, 11:09am UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/9 "2019-10-25T11:09:14Z")

</div>

Hi @gil, after several times I configured APM server and now can push data to elasticsearch.  
I figured out that environment variables in deployment.yaml file didn't work for me ( **container logs in Azure** were rather informative) and apm-server.yml file on host is existed in readonly mode. I couldn't edit it either.

So I looked at Narendiran D post ([https://medium.com/logistimo-engineering-blog/how-did-i-use-apm-in-kubernetes-ecosystem-8f22d52beb03](https://medium.com/logistimo-engineering-blog/how-did-i-use-apm-in-kubernetes-ecosystem-8f22d52beb03)) and configured **ConfigMap** object (all environment variables were migrated to ConfigMap).

After these manipulations everything works fine.

---

<div class="post-metadata">

**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\
**Post date:** [October 25, 2019, 12:15pm UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/10 "2019-10-25T12:15:25Z")

</div>

Glad to hear it, thanks for following up.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2019, 8:15am UTC](https://discuss.elastic.co/t/couldnt-verify-apm-server-status/202511/11 "2019-11-15T08:15:27Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.