# APM Server Cluster Monitoring in Kibana

**URL:** <https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587>\
**Category:** APM\
**Tags:** elastic-stack-monitoring\
**Created:** [January 17, 2019, 8:27am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587 "2019-01-17T08:27:32Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 17, 2019, 8:27am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/1 "2019-01-17T08:27:32Z")

</div>

Hi,

We've set up an Elastic APM cluster composed of three APM servers fronted by Haproxy.

Load testing result has been great on this stack, and I could see that all APM Server instances are happily distributing the load against a cluster of seven Elasticsearch nodes (3 master only, 4 data only). I confirm this in the Haproxy response headers.

One thing I noticed is that Kibana monitoring is only showing a single APM Server instance, which I suspect is the first APM server to register in Elasticsearch. **How do i get the other APM Server instances to display in Kibana?** APM Servers count is "1"

Our set up:

```
Haproxy ----> (3) APM Servers ----> (7) Elasticsearch <---Kibana
```

All are installed via docker.

I've tested two configuration for the APM Servers: via individually started containers and as a docker swarm scale up. Results are still the same, only one APM Server instance is displayed in the Kibana monitoring page (using Docker container ID). APM server containers are all built on the same image, all share the same configuration except for the individually started containers which have their own published IP address being used by Haproxy.

Kibana correctly shows 7 Elasticsearch server instances.

[Edit]  
**Forgot to mention that Machine Learning shows three beat names corresponding to the docker ids of the APM Server containers.**

Any help is appreciated **as we'd like to be able to monitor all APM Server instances in Kibana.**  
[Edit]

Thanks!

Ronald

---

<div class="post-metadata">

**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\
**Post date:** [January 17, 2019, 4:35pm UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/2 "2019-01-17T16:35:25Z")

</div>

@digitalron I'm glad to the hear the load testing is going well. I'm only able to reproduce your results when setting the `name` directive in `apm-server.yml` - I see in your edit ML is showing three beat names but I'm hoping you can confirm those are not set or unique across your apm-servers? The Kibana UI essentially does this query:

```auto
GET .monitoring-beats-6-*/_search
{
  "size": 0, 
  "aggs": {
    "apm-servers": {
      "terms": {
        "field": "beats_state.beat.name",
        "size": 10
      }
    }
  }
}

```

If that returns multiple entries then something is indeed wrong. An aggregation on `beats_state.beat.host` should show all apm-servers regardless of that setting, so you can compare the results of those queries to be sure each server is reporting in.

If everything checks out, can you provide more details about your installation including Elastic Stack component versions and relevant apm-server.ymls?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 17, 2019, 9:36pm UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/3 "2019-01-17T21:36:06Z")

</div>

Beats typically generate an identifier when they first start up, so if you simply clone this you can end up with multiple instances that have the same I’d and look like one.

---

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 18, 2019, 7:40am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/4 "2019-01-18T07:40:27Z")

</div>

Thanks Gil and Christian for the responses. I double checked the settings and ran the query above pon both instances and got the following results:

**3 individually created APM Server containers**  
One APM Server found in Monitoring  
APM Server Name in Monitoring is the same as container id  
Three beats ids found in ML, corresponding to their respective docker container ids  
Dev Tools Query in Kibana returned a single beat\_state, hoist and name corresponded to docker container and the same displayed in monitoring

**3 docker swarm scaled APM Server containers**  
One APM Server found in Monitoring  
APM Server Name in Monitoring is the same as container id  
Three beats ids found in ML, corresponding to their respective docker container ids  
Dev Tool Query in Kibana **did not** return beats\_state property. It did have **beat\_stats** but like the one above returned a single host and name that corresponded to the docker id displayed under monitoring

Both used the same docker image for APM Server 6.5.4  
I did not set any name, and spinning down the containers and spinning up a new one results in the same setup but with a different APM server name which points to a dynamic naming based on the docker container id.

Additional inputs:  
The individually created APM Server Docker containers are assigned static ports to which Haproxy points to. The swarm scaled one uses the default 8200 port for each container.

Thanks!

Ronald

---

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 18, 2019, 7:49am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/5 "2019-01-18T07:49:00Z")

</div>

Related screenshots

**Individually created containers**

 ![30](https://us1.discourse-cdn.com/elastic/original/3X/3/8/380a90368898ae58ec83cf538c51430214cafa44.png)  
 ![36](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b0b7d062b97a60d275a487b3bcc2b62ae128ff63.png)

**Docker Swarm-scaled containers**

 ![04](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2d37e5b33d85814b657b458c468709497f1292be.png)  
 ![07](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b75455c72ec0602602c9c73573e1de1e18d210d6.png)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 18, 2019, 7:50am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/6 "2019-01-18T07:50:46Z")

</div>

So you are launching them directly based off the official base image?

---

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 18, 2019, 7:51am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/7 "2019-01-18T07:51:55Z")

</div>

Yes. Got them from [www.docker.elastic.co](http://www.docker.elastic.co)

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [January 18, 2019, 6:32pm UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/8 "2019-01-18T18:32:24Z")

</div>

Hi @digitalron,

Could you run the following query please?

```auto
POST .monitoring-beats*/_search
{
  "size": 0,
  "aggs": {
    "by_uuid": {
      "terms": {
        "field": "beats_stats.beat.uuid",
        "size": 10
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 19, 2019, 6:17am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/9 "2019-01-19T06:17:15Z")

</div>

Hi Shaunak, here is the result of running the query:

```
{
  "took" : 25,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 12339,
    "max_score" : 0.0,
    "hits" : []
  },
  "aggregations" : {
    "by_uuid" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "8a6e4932-c161-4d82-a192-fc532a04354b",
          "doc_count" : 10560
        }
      ]
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 21, 2019, 10:30am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/10 "2019-01-21T10:30:18Z")

</div>

Some additional findings: I observed the monitoring page for some time and noticed that once in a while, the node being displayed changes:

One moment:

 ![13](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3da2b8301769cf6a21557ea850262f8f693fd5c5.png)

After a few seconds...

 ![46](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0c673714a6e9270d41b8e1fd79954b3bb7abcd5c.png)

There is no pattern in how the displayed node changes each time the screen refreshed. One node can be displayed for four successive refreshes, then it alternates, then the other would be displayed thrice in succession, then they would alternate again, seemingly at random.

---

<div class="post-metadata">

**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\
**Post date:** [January 22, 2019, 10:07am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/11 "2019-01-22T10:07:47Z")

</div>

Okay, so I cleared everything and started from scratch, re-pulled the 6.5.4 Docker images and re-ran docker-compose, and the three APM servers are now showing in the monitoring page.

I have no idea what happened but it seems to work so I guess in such a situation, it's probably better to just create new APM Server container instances.

This can be closed now. Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 12, 2019, 6:08am UTC](https://discuss.elastic.co/t/apm-server-cluster-monitoring-in-kibana/164587/12 "2019-02-12T06:08:54Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.