Currently all the beats and logstash sends data directly to Elasticsearch.
I have multiple servers installed with heartbeat,metricbeat,logstash and filebeat and now I've requirement to monitor all these components stats in kibana dashboard.
ex: on which host running, status up or down, filebeat/logstash ingestion time,harvester info,etc.
In short I need to monitor all components stats depoyed in multiple servers and view in kibana dashboard.
Plz suggest how this can be done for above components.
You have two main choices: Kibana Dashboard or the Stack Monitoring UI in Kibana.
In either case, you'll need to deploy some collector to collect the stats. You can use Elastic Agent with the various Integration for the Elastic products or use Metricbeat and Filebeat to collect the metrics and logs with the right modules for each Elastic product.
Taking Metricbeat as an example, you would then need either Elastic Agent and the Beat Integration or Metricbeat/Filebeat with the Beat module enabled.
You'll have to configure the production Metricbeat to expose it's metrics endpoint and then make the monitoring Metricbeat or Elastic Agent fetch data from that endpoint.
The production Metricbeat will ship data to your production Elasticsearch cluster, while the monitoring Metricbeat will ship data to (preferably) a dedicated monitoring Elasticsearch cluster.
If you chose to use the Stack Monitoring UI, you'll need to use the X-pack module versions.
If you choose to use Kibana Dashboards, you can use the non X-pack versions but will then need to build the dashboards yourself.
Thanx for your response.I'm using ELK 8.8.2 version Enterprise license
What I tried is following:
The servers where Metricbeat,Heartbea,Filebeat and Logsatsh are installed in there respective YML file i added below hoping that this will show data in stack monitoring.
monitoring.enabled: true
I'm not sure after reading documentaion waht exact settings should be added to monitor these components in stack monitoring, does kibana also require some settings?
I have both requirement to show in stack monitoring and kibana dashboard also. I don't see clear implementaion how this can be done. Could you please guide with relevant settings for 8.8.2 version?
I went through few community comments for my implementation requirements.I'm not sure of exact ans and would request to guide on same.
I migrated from ELK 7.9.1 version to 8.8.2 version. Earlier in 7.9.1 version under stack monitoring I could see everything(metricbeat,heartbeat,filebeat,logstash STATS) and same trying to achieve in 8.8.2.
I read in one of the community comments that Self monitoring is depriciated? ,is it true ,like in kibana itself we no longer have functionality to monitor our stack?
I'm not sure what exact settings should I do and where so that I can monitor my exact stack.
Now, As per my understading if stack monitoring in kibana itself(self monitoring) is depriciated then only way is to monitor through metricbeat only?
I have multiple beats-Metricbeat,Heartbeat,Filebeat deployed in multiple servers and I'am not getting exact settings what/where to use so that metricbeat can monitor these components stats like stack monitoring. The final desired outcome is to monitor both stack monitoring in kibana and also in dashboards.
The following stats ex:
Metricbeat- status-up or down, host server,cpu and ram used
Filebeat- Harvester active, current ingestion stats, status-up or down,cpu and ram used etc
Same goes for logstash and heartbeat.
elasticsaerch,kibana yml files require any settings ,if yes then what are those? , Simlilarly in metricbeat and other beats and logsatsh YML what settings should be used or everything if through metricbeat can be done then HOW?
Not exposed to xpack and through documentation finding hard time.
I'm not sure or exposed to xpack, What are and where these xpack setting should be so that I can monitor all my stack componenets in dashboard and kibana UI.
Please recommend relevant settings.
We make a distinction between self monitoring and dedicated monitoring. This refers to if your monitoring data is collocated with your production data in the same Elasticsearch cluster or if you have a dedicated Elasticsearch cluster for your monitoring data. We strongly recommend using dedicated monitoring so that your monitoring data is reachable even if your production cluster is facing issues.
Then we make another distinction between internal collection and external collection. Internal collection refers to the legacy collection systems built into each stack component, which are under way of being deprecated though you can still freely use them if they suit your needs better. External collection means using another component to monitor, so you could use Metricbeat or Elastic Agent. In the long run, we strongly recommend using external collection.
Note that the Stack Monitoring UI supports all the combinations of storage location and collection mode.
Note also that those choices are also supported by Kibana Dashboards.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.