Stack Monitoring with Fleet/elastic-agent

In Kibana, when you go to Stack Monitoring, it says "No monitoring data found" and suggests using Metricbeat. Except, shouldn't we be using Elastic Agent?

So, how can I get the Stack Monitoring page working with Agent? (configuring the Elasticsearch Integration in Fleet didn't seem to be enough).

1 Like

Hi @rastro, I'm definitively interested in this too!
Didn't find any answer and same conclusion about Fleet.

Stack monitoring should work out of the box when using the correct integrations. Make sure to look for the elastic-agent integrations only, not the beats integrations. Those won't work.

Also make sure that in the integration itself you set the stack-monitoring option enabled and proper authentication is set in the option.
When all that is done, also make sure that in the application itself (be it Kibana, Elastic or Logstash) the monitoring endpoint is configured and accessible.

Hope this will help.

1 Like

Thanks @fselim,

You pushed me on the right tracks!

So I've found this Elastic documentation which seems to be perfectly clear :

I will follow it and give you a feedback.
Many thanks!

I believe that I've followed those directions before, even trying both the user auth and API key configs, but I still get nothing shown in KIbana.

(tested that the username and password can connect to localhost on each node)
scope: node

Hi @rastro and @fselim,

For the moment, I'm still unable to get Stack Monitoring working with Elastic Agent...
Here is what I've done until now :

  • Installed Elastic Agent on Master and Data nodes with a specific policy including "Elasticsearch" integration.
  • Created an API key for the "elastic" user and use it for the Elasticsearch integration
  • Installed Elastic Agent on Kibana node with a specific policy including "Kibana" integration
  • Everything is fine and Healthy in Fleet.
  • Created a specific user named "kibana_monitoring" with the role monitoring_user.
  • Followed Collecting Elasticsearch monitoring data with Elastic Agent
  • Followed View monitoring data in Kibana

Here is my kibana.yml content :

monitoring.ui.enabled: true
monitoring.ui.elasticsearch.username: "kibana_monitoring"
monitoring.ui.elasticsearch.password: "REDACTED"

One question please, should I activate Stack Monitoring by clicking on "Or, set up with self monitoring" ?

I'm not sure about that.

To be honest, I'm pretty stuck at the moment...

My actual setup is :

  • 3x dedicated ES Master nodes
  • 1x Kibana node
  • 1x Fleet server node
  • ES Hot, Warm and Cold nodes

Thanks a lot for your help.

So, should I set up with self monitoring or not?

Of course I forgot to mention that the "elasticsearch" integration is working fine and the associated Dashboards are fine too.

What Version?

Did you go through

and no you should not do Self Monitoring...

Not sure you needed to do the following...

monitoring.ui.enabled: true <!-- This is the default 
monitoring.ui.elasticsearch.username: "kibana_monitoring"
monitoring.ui.elasticsearch.password: "REDACTED"

These settings adjust how Stack Monitoring displays monitoring data. However, the defaults work best in most circumstances. For more information about configuring Kibana, see Setting Kibana server properties.

For debug try the elastic user/pw first see if that works...
then if it does go back and look at the user / roles...

Per here

  • If the Elastic security features are enabled, expand Advanced options under the Hosts setting and enter the username and password of a user that has the remote_monitoring_collector role.
1 Like

So there are two roles mentioned in the doc that I can see:

  • remote_monitoring_collector (in "Collecting Elasticsearch monitoring data with Elastic Agent") which appears to be for collecting and sending the data.
  • monitoring_user (in "View monitoring data in Kibana") which is used by the Kibana integration to "collect data about Kibana". This is what your reply focused on, but the original request was about monitoring the cluster, not Kibana.

The monitoring_user role is also mentioned in "View monitoring data in Kibana", where a user with that role should be configured as monitoring.ui.elasticsearch.username, which defaults to elasticsearch.username, which is commented out by default but suggested as "kibana_system".

If you go into the UI and try to reset the password for kibana_system (so you can hardcode the password as elasticsearch.password in kibana.yml), it warns: "KIbana will lose connection to Elasticsearch". As Kibana is currently connecting just fine to elasticsearch to browse other data, it seems like this does not need to be reset or hardcoded in kibana.yml.

So, now what?

Hi @rastro I am not sure what you are referring to I think you are confusing several topics.

kibana_system is the user that Kibana uses to connect to Elasticsearch, it has nothing to do with the monitoring topic / capabilities... and yes if you change the password through the UI it will break the connection to Elasticsearch

I believe @DaddyYusk Is asking about monitoring Kibana as a component with the Elastic Stack Monitoring (as he seems to already have the cluster monitoring working), which is what my response is focused on. Perhaps I have misunderstood.

So you got it working? What did you do?

I was referring to step 3 of View monitoring data in Kibana | Kibana Guide [8.11] | Elastic, where it talks about a user with monitoring_user privs, and setting it in kibana.yml. Tracking back those config params led me back to the kibana_system user, whose password i clearly should not need to change.

So, I still have the original problem of not being able to view cluster information when I go to Stack Monitoring.

Ahh Sorry... I was answering the other User...

What version are you on?

I am not clear exactly what you have or have not tried...

Are you trying to set up monitoring with

  1. metricbeat or
  2. Elastic Agent

you need to choose 1 or the other... which one do you want to focus on?

Are you sending the monitoring data to

A) the same cluster that you are monitoring? or
B) a separate dedicated monitoring cluster?

Help me understand what you are trying to do... And the version and perhaps I can help

I think the docs sometimes can be a bit confusing...

Version 8.11.0

As the post subject says, I'm trying to set up Stack Monitoring using elastic-agent.

The data is going to the same cluster.

I should have been able to go to Stack Monitoring and been provided information on the "right way" to set this up (i.e. elastic-agent), but that page gave information on two "wrong ways" (metricbeat and manually).

So, I found this doc: Collecting Elasticsearch monitoring data with Elastic Agent | Elasticsearch Guide [8.11] | Elastic

and I believe I have followed all those steps to release an Elasticsearch integration to the nodes. I can see no indication that this integration is failing.

The last step (labeled "View monitoring data in Kibana", which sort of makes you think that Kibana would show you data now, but really should be "Configuring Kibana to view monitoring data") takes you here: View monitoring data in Kibana | Kibana Guide [8.11] | Elastic
and, from what I read, everything is supposed to default to a working value.

So, back to Stack Monitoring, where you still just get the message on how to set things up incorrectly.

The doc is seemingly written as a reference for the people who wrote the code, which really doesn't serve the rest of the universe very well. I can only imagine how elastic noob must feel!

Hope the extra info helps.


Apologies I got distracted with the other user...

If I get a chance (since we are all just volunteers here), I will take a look end-to-end.

Sorry, you are finding it frustrating... the docs definitely could be better on this, I think it is the result of the evolution and still supporting past methods... it gets kinda jumbled.

I'll take a look when I get a chance .. in the past, for monitoring to the same cluster, I never needed to update the kibana.yml roles etc,

I will do a fresh install and see what I see...

1 Like

Ok I installed Elasticsearch and Kibana, Elastic Agent 8.11.1, on a single UbuntuHost
All OOTB Security, All Defaults, and enrolled Kibana, this is self-signed certs.

I made no changes to elasticsearch.yml

In kibana.yml I set / added these ""
xpack.encryptedSavedObjects.encryptionKey: "askdjfh-2287346-laksdjfhaksfdjh-387246523984756"

Those are the only mods I made.

Then I added the fleet server I with these settings.... is the IP of the host
I got this command from the Kibana -> Fleet and Agent -> Add Fleet Server shown here

sudo ./elastic-agent install \
   --fleet-server-es= \
   --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MDEzMTY5MDkxMjM6bmItaHNDX0xTVkd5NDBUVGNtemxMUQ \
   --fleet-server-policy=fleet-server-policy \
   --fleet-server-es-ca-trusted-fingerprint=8d4d9a8c78ac6d986b0f3d2c971851afdf3b8cf593f5571e78f64f29545523a7 \

Then I just added the Elastic Integration and used and API Key
The integration needs the CA so I just made it readable by all and that is what I set in the integration

# ls -l /etc/elasticsearch/certs/http_ca.crt 
-rw-rw-r-- 1 root elasticsearch 1915 Nov 29 20:42 /etc/elasticsearch/certs/http_ca.crt

In the SSL configurations

certificate_authorities: ["/etc/elasticsearch/certs/http_ca.crt"]

The Agent and Policy Loaded Up...

And the Elasticsearch Cluster Shows up (no other changes to the elasticsearch.yml or kibana.yml

So that works... I will work on Kibana Next..


I added the Kibana Integration using these instructions

No other changes to kibana.yml

I created a user as specified in the directions:

user: kibana_monitoring
role: remote_monitoring_collector

Then I added the Kibana Integration

Applied to the same Agent

And Now Kibana Shows in stack monitoring...

So in Summary for Elasticsearch and Kibana Monitoring with Agent.

  • I only made 2 changes to the kibana.yml In the post above, no other changes to any .yml ""
xpack.encryptedSavedObjects.encryptionKey: "askdjfh-2287346-laksdjfhaksfdjh-387246523984756"
  • To monitor elasticsearch with the agent just needed, the correct URL, and API key in beats format, and the CA to be reachable.See Settings Above

  • To monitor kibana I just created a user with single correct roles and the correct URL
    user: kibana_monitoring
    role: remote_monitoring_collector

1 Like

Hi @stephenb and @rastro,

First things first, many thanks for all your answers!
Just to make it clear :

  • I have the exact same problem as @rastro
  • My Elastic Agent version is 8.10.4
  • I've followed all your last 2 posts @stephenb for both "elasticsearch" and "kibana" integration
  • At the moment, I'm still unable to have Stack Monitoring working.

My actual setup is :
- 3x dedicated ES Master nodes
- 1x Kibana node
- 1x Fleet server node
- ES Hot, Warm and Cold nodes
- No monitoring cluster! I try to have Stack Monitoring enabled on the same production cluster

But something differs for my setup :

As I have dedicated master nodes but no load-balancing proxy fronting, I'm not sure about the proper setup for the Scope.

Thanks a lot for your help!

Hi @DaddyYusk

I want to be respectful of @rastro Since They are the Orginal Poster.

We do not know that because they have not responded after my instructions.

  • What exactly is not working?
  • Exactly What errors do you have?
  • What other changes have you made to your elasticsearch.yml and / or kibana.yml
  • If you have already pressed/selected self-monitoring, that makes changes that might need to be undone.
  • Are you sure that ca-cert is available and can be read by the agent?
  • I can not help really with just "It does not work"
  • Have you tried logging into where they agent is and running status or logs command
  • Have you tried uninstalling and reinstalling the agent?

All that language is so that you do not request the monitoring data from a dedicated master node, which should not be burdened with these types of requests.

Your choices are

  1. User node and add every non-dedicated master node.... (painful but technically best practice)

  2. Use cluster and point it to one of your hot nodes (this may cause a small amount of load to that node, usually pretty small) because, in the end, every node is actually an endpoint to the entire elasticsearch cluster.