Hello,
Essentially, I am looking to monitor when my server is getting full, using metricbeat to log the space left on the filesystem, and also using a filebeat cronjob to monitor more specifically where all of the storage is coming from, should the server's capacity near the point of being full.
I am currently running two beats:
1 Metricbeat - system module, filesystem metricset, in order to see the total amount of space left on a given filesystem.
1 Filebeat - tails a local file that is appended to by a cronjob, which calls "du" on a specific directory to get info on how much storage each of the subdirectories are taking up within that directory.
My main goal is to be able to send out an alert once metricbeat detects that the total space available left on the machine goes below a certain threshold. In that alert, I would also like to show the subdirectories' from the filebeat's storage consumption, so that I can monitor which of the directories is likely to be the culprit of the filesystem nearing max storage capacity.
Example:
system.filesystem.available: <some amount of total space left on server>
subdirectory_1: <taking up this much space>
subdirectory_2: <taking up this much space>
subdirectory_3: <taking up this much space>
My idea is to just grab one field from the metricbeat, the one that shows how much space is available in the filesystem, and append that field to a filebeat event that would have been logged at around the same time. I would be doing this so that I would have the field that shows how much space is left on the server, and the fields that display how much space each of these directories I am monitoring with the cronjob is taking up, all in one event.
You could say I am trying to take part of the functionality of Metricbeat (giving total space available in a given filesystem), and just tacking it onto filebeat.
Of course, since the log from the metricbeat system module, filesystem metricset would look totally different from a filebeat log that just tails a file, there would be no common fields that are also unique to each pair of events that I would want to combine. Therefore I don't see how I can effectively use the logstash "aggregate" filter for this use case. Is there any way I can solve this?
Maybe there is a more elegant way to begin with to achieve getting this data in one event using beats/logstash? Any help is appreciated! Thank you!
PS: I use beats and logstash for log collecting and parsing, but opensearch for alerting, in case that is a necessary variable worth mentioning
)