In Kibana, when you go to Stack Monitoring, it says "No monitoring data found" and suggests using Metricbeat. Except, shouldn't we be using Elastic Agent?
So, how can I get the Stack Monitoring page working with Agent? (configuring the Elasticsearch Integration in Fleet didn't seem to be enough).
Hi @rastro, I'm definitively interested in this too!
Didn't find any answer and same conclusion about Fleet.
Stack monitoring should work out of the box when using the correct integrations. Make sure to look for the elastic-agent integrations only, not the beats integrations. Those won't work.
Also make sure that in the integration itself you set the stack-monitoring option enabled and proper authentication is set in the option.
When all that is done, also make sure that in the application itself (be it Kibana, Elastic or Logstash) the monitoring endpoint is configured and accessible.
Hope this will help.
Thanks @fselim,
You pushed me on the right tracks!
So I've found this Elastic documentation which seems to be perfectly clear :
I will follow it and give you a feedback.
Many thanks!
I believe that I've followed those directions before, even trying both the user auth and API key configs, but I still get nothing shown in KIbana.
https://localhost:9200
username/password
(tested that the username and password can connect to localhost on each node)
scope: node
For the moment, I'm still unable to get Stack Monitoring working with Elastic Agent...
Here is what I've done until now :
monitoring_user.Here is my kibana.yml content :
monitoring.ui.enabled: true
monitoring.ui.elasticsearch.username: "kibana_monitoring"
monitoring.ui.elasticsearch.password: "REDACTED"
One question please, should I activate Stack Monitoring by clicking on "Or, set up with self monitoring" ?
I'm not sure about that.
To be honest, I'm pretty stuck at the moment...
My actual setup is :
Thanks a lot for your help.
Regards.
So, should I set up with self monitoring or not?
Of course I forgot to mention that the "elasticsearch" integration is working fine and the associated Dashboards are fine too.
What Version?
Did you go through
and no you should not do Self Monitoring...
Not sure you needed to do the following...
monitoring.ui.enabled: true <!-- This is the default
monitoring.ui.elasticsearch.username: "kibana_monitoring"
monitoring.ui.elasticsearch.password: "REDACTED"
These settings adjust how Stack Monitoring displays monitoring data. However, the defaults work best in most circumstances. For more information about configuring Kibana, see Setting Kibana server properties.
For debug try the elastic user/pw first see if that works...
then if it does go back and look at the user / roles...
Per here
- If the Elastic security features are enabled, expand Advanced options under the Hosts setting and enter the username and password of a user that has the
remote_monitoring_collectorrole.
So there are two roles mentioned in the doc that I can see:
The monitoring_user role is also mentioned in "View monitoring data in Kibana", where a user with that role should be configured as monitoring.ui.elasticsearch.username, which defaults to elasticsearch.username, which is commented out by default but suggested as "kibana_system".
If you go into the UI and try to reset the password for kibana_system (so you can hardcode the password as elasticsearch.password in kibana.yml), it warns: "KIbana will lose connection to Elasticsearch". As Kibana is currently connecting just fine to elasticsearch to browse other data, it seems like this does not need to be reset or hardcoded in kibana.yml.
So, now what?
Hi @rastro I am not sure what you are referring to I think you are confusing several topics.
kibana_system is the user that Kibana uses to connect to Elasticsearch, it has nothing to do with the monitoring topic / capabilities... and yes if you change the password through the UI it will break the connection to Elasticsearch
I believe @DaddyYusk Is asking about monitoring Kibana as a component with the Elastic Stack Monitoring (as he seems to already have the cluster monitoring working), which is what my response is focused on. Perhaps I have misunderstood.
So you got it working? What did you do?
I was referring to step 3 of View monitoring data in Kibana | Kibana Guide [8.11] | Elastic, where it talks about a user with monitoring_user privs, and setting it in kibana.yml. Tracking back those config params led me back to the kibana_system user, whose password i clearly should not need to change.
So, I still have the original problem of not being able to view cluster information when I go to Stack Monitoring.
Ahh Sorry... I was answering the other User...
What version are you on?
I am not clear exactly what you have or have not tried...
Are you trying to set up monitoring with
you need to choose 1 or the other... which one do you want to focus on?
Are you sending the monitoring data to
A) the same cluster that you are monitoring? or
B) a separate dedicated monitoring cluster?
Help me understand what you are trying to do... And the version and perhaps I can help
I think the docs sometimes can be a bit confusing...
Version 8.11.0
As the post subject says, I'm trying to set up Stack Monitoring using elastic-agent.
The data is going to the same cluster.
I should have been able to go to Stack Monitoring and been provided information on the "right way" to set this up (i.e. elastic-agent), but that page gave information on two "wrong ways" (metricbeat and manually).
So, I found this doc: Collecting Elasticsearch monitoring data with Elastic Agent | Elasticsearch Guide [8.11] | Elastic
and I believe I have followed all those steps to release an Elasticsearch integration to the nodes. I can see no indication that this integration is failing.
The last step (labeled "View monitoring data in Kibana", which sort of makes you think that Kibana would show you data now, but really should be "Configuring Kibana to view monitoring data") takes you here: View monitoring data in Kibana | Kibana Guide [8.11] | Elastic
and, from what I read, everything is supposed to default to a working value.
So, back to Stack Monitoring, where you still just get the message on how to set things up incorrectly.
The doc is seemingly written as a reference for the people who wrote the code, which really doesn't serve the rest of the universe very well. I can only imagine how elastic noob must feel!
Hope the extra info helps.
Apologies I got distracted with the other user...
If I get a chance (since we are all just volunteers here), I will take a look end-to-end.
Sorry, you are finding it frustrating... the docs definitely could be better on this, I think it is the result of the evolution and still supporting past methods... it gets kinda jumbled.
I'll take a look when I get a chance .. in the past, for monitoring to the same cluster, I never needed to update the kibana.yml roles etc,
I will do a fresh install and see what I see...
Ok I installed Elasticsearch and Kibana, Elastic Agent 8.11.1, on a single UbuntuHost
All OOTB Security, All Defaults, and enrolled Kibana, this is self-signed certs.
I made no changes to elasticsearch.yml
In kibana.yml I set / added these
server.host: "0.0.0.0"
xpack.encryptedSavedObjects.encryptionKey: "askdjfh-2287346-laksdjfhaksfdjh-387246523984756"
Those are the only mods I made.
Then I added the fleet server I with these settings.... 10.168.0.12 is the IP of the host
I got this command from the Kibana -> Fleet and Agent -> Add Fleet Server shown here
sudo ./elastic-agent install \
--fleet-server-es=https://10.168.0.12:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MDEzMTY5MDkxMjM6bmItaHNDX0xTVkd5NDBUVGNtemxMUQ \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca-trusted-fingerprint=8d4d9a8c78ac6d986b0f3d2c971851afdf3b8cf593f5571e78f64f29545523a7 \
--fleet-server-port=8220
Then I just added the Elastic Integration and used and API Key
The integration needs the CA so I just made it readable by all and that is what I set in the integration
# ls -l /etc/elasticsearch/certs/http_ca.crt
-rw-rw-r-- 1 root elasticsearch 1915 Nov 29 20:42 /etc/elasticsearch/certs/http_ca.crt
In the SSL configurations
certificate_authorities: ["/etc/elasticsearch/certs/http_ca.crt"]
The Agent and Policy Loaded Up...
And the Elasticsearch Cluster Shows up (no other changes to the elasticsearch.yml or kibana.yml
So that works... I will work on Kibana Next..
I added the Kibana Integration using these instructions
No other changes to kibana.yml
I created a user as specified in the directions:
user: kibana_monitoring
role: remote_monitoring_collector
Then I added the Kibana Integration
Applied to the same Agent
And Now Kibana Shows in stack monitoring...
So in Summary for Elasticsearch and Kibana Monitoring with Agent.
server.host: "0.0.0.0"
xpack.encryptedSavedObjects.encryptionKey: "askdjfh-2287346-laksdjfhaksfdjh-387246523984756"
To monitor elasticsearch with the agent just needed, the correct URL, and API key in beats format, and the CA to be reachable.See Settings Above
To monitor kibana I just created a user with single correct roles and the correct URL
user: kibana_monitoring
role: remote_monitoring_collector
First things first, many thanks for all your answers!
Just to make it clear :
My actual setup is :
- 3x dedicated ES Master nodes
- 1x Kibana node
- 1x Fleet server node
- ES Hot, Warm and Cold nodes
- No monitoring cluster! I try to have Stack Monitoring enabled on the same production cluster
But something differs for my setup :
As I have dedicated master nodes but no load-balancing proxy fronting, I'm not sure about the proper setup for the Scope.
Thanks a lot for your help!
Regards
Hi @DaddyYusk
I want to be respectful of @rastro Since They are the Orginal Poster.
We do not know that because they have not responded after my instructions.
All that language is so that you do not request the monitoring data from a dedicated master node, which should not be burdened with these types of requests.
Your choices are
User node and add every non-dedicated master node.... (painful but technically best practice)
Use cluster and point it to one of your hot nodes (this may cause a small amount of load to that node, usually pretty small) because, in the end, every node is actually an endpoint to the entire elasticsearch cluster.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.