Create custom visualization dashboard from the monitoring data?

[Mattness]

Hello, I noticed all the graphs under Stackmonitoring/elasticsearch/nodes and was wondering where Kibana gets this data from. Also can I access this data myself and create custom visualizations for my dashboard? Or is the dashboard/discover area only for external log data?

I am fairly new to Elastic and Kibana as you can probably tell and I am trying to get a sense of what my possibilities are. Any and all pointers are very much appreciated.

[mikecote (Mike)]

Depending on the data you're looking for, some is pulled directly from Elasticsearch APIs (https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html) like the list of nodes and their current health. For historical data and to be able to create visualizations on it, you can do so with metric beats (https://www.elastic.co/beats/metricbeat) that collect and store data within an Elasticsearch index that you can then load up in Kibana and do some visualizations with.

Hope this helps!

[Mattness]

Thank you very much for your response
I am looking for statistical data about my cluster, nodes and indices. E.g. I want to display my docs with the most attachments, # of items grouped by category ... that kind of thing. Can I pull this from ElasticSearch via metric beats and then send it via logstash to kibana for visualization?

[mikecote (Mike)]

Ah, there's queries for each of these scenarios.

I want to display my docs with the most attachments

For this, if you have attachments stored as a number, you can use the search API and sort by that field in a descending order Search API | Elasticsearch Guide [8.11] | Elastic

number of items grouped by category

For this, you can add an aggregation to your search query Terms aggregation | Elasticsearch Guide [8.11] | Elastic.

The link below will show you all the documentation that there is for Elasticsearch including different types of queries you can do, different types of stats you can do and such Elasticsearch Guide [8.11] | Elastic.

[Mattness]

Thanks again, I think I named the wrong kind of queries. I already knew about the search queries.
I have so far been playing around with Winlogbeat and Logstash. If I may refer to my original question ... can I gather all the data necessary to make a "custom" stack monitoring via metricbeat? E.g. JVM Heap, Index Memory, I/O Operations Rate, etc. are the kinds of visuals I want to put together.

TL;DR I want to know where xpack.monitoring.collection.enabled gets the data from when set to "true".