APM Server monitoring visualizations in Kibana

[AlessandroKP (Alessandro)]

Hi all,

I have a question regarding APM and the Stack Monitoring feature of Kibana.

I have an ES cluster with different components attached to it: beats, a Kibana instance, a Logstash instance with many pipelines and an APM server.
The output of the APM server is not sent to ES directly, but it is handled by a Logstash pipeline which applies some filters and then outputs the results to the ES cluster.
When I go to the Kibana section "Stack Monitoring", there is a record for my ES cluster.
When I click on it I can see the informations about all the ES nodes, the Kibana instances, the beats that are sending their output to ES, the Logstash nodes with all their pipelines, but the APM server is not present.
In fact, in the Stack Monitoring page I get this message: "It looks like you have instances that aren't connected to an Elasticsearch cluster".
In the table below, in addition to the ES cluster, there is an other record for a cluster named "Standalone cluster" which is exactly the APM server (see screenshot below).

When I click on "Standalone cluster" I get an overview of the APM server and the nodes forming the APM cluster (see screenshot below).

Now, I would like these informations to be included in the ES cluster overview, as if the APM server was actually a component directly connected to the ES cluster (like Kibana, Logstash and the beats).
This is the case if the APM server output is sent directly to ES, but not if the output is taken by Logstash.

So the question is:
is there a way (maybe, by changing some configuration setting related to the APM server monitoring) to make the APM server appear as a component of the ES cluster in the Stack Monitoring panel of Kibana, even if the output of the APM is handled by Logstash, and not as a standalone cluster?

[Igor_Zaytsev]

Hey @AlessandroKP,

You still need to enable monitoring for that cluster: Use internal collection to send monitoring data | APM User Guide [master] | Elastic take a look at #2

If you configured a different output, such as Logstash or you want to send APM Server monitoring events to a separate Elasticsearch cluster

[AlessandroKP (Alessandro)]

Hi Igor,

I had already enabled monitoring by following the steps in https://www.elastic.co/guide/en/apm/server/master/monitoring-internal-collection.html,
as you suggested. However, this did not solve my issue.

This is what I have in the configuration file for the apm server:

apm-server:
  host: "localhost:8200"

output.logstash:
  enabled: true
  hosts: ["localhost:5052"]

monitoring:
  enabled: true
  elasticsearch:
    hosts: ["localhost:9200"]
    username: apm_system
    password: apm_system

As I wrote in the previous post, with this configuration the APM server is not shown as being part of the main Elastic cluster in the Monitoring UI,
but as a standalone cluster.

[Igor_Zaytsev]

Hey @AlessandroKP

Currently there is no way to figure out cluster_uuid if the output is anything other than elasticsearch. If there is no cluster_uuid we mark it as standalone.

We are currently trying to improve this design here: https://github.com/elastic/beats/pull/13251 Feel free to leave any suggestions/feedback