Hi Everybody. We're new to ELK but not new to SIEMs. We're trying to improve our visibility by creating something in the ELK stack that might be considered "Visibility Monitoring". Our "Visibility Monitoring" comes with two objectives:
1) Identify any new log source (a new server, a new device, a new cloud
service, etc) that has been added to the company's environment which
ELK is currently not collecting-from. (So it can be added for SIEM log
collection to improve Visibility) AKA - New Log Source Detection
2) Identify any existing log source (a server, a device, a cloud service, etc)
that was initially successfully set-up for log collection in ELK, but now ELK
is failing to obtain logs from that same specific log source. (So the issue
with collecting logs from that specific source can be identified and
resolved to improve Visibility.) AKA - Failed Log Source Detection
In both cases, for "New Log Source Detection" and "Failed Log Source Detection", we would use scripts that would periodically gather information, from various resources in our environment, and create different lists for each log source type that would be written to separate files. These different files would be considered separate baselines for each log source type (list of servers, list of devices, list of cloud services, etc).
Then, we want to compare what's contained in a given file (a list of servers, a list of devices, or a list of cloud services, etc) to what is actually being ingested by ELK and produce a Visualization that would display what ELK is currently missing in log collection.
Given all of this, how do we have Kibana ingest a file (a list of servers, a list of devices, or a list of cloud services, etc) that can be used in Kibana to compare-to what is actually being ingested in ELK and produce a Visualization that displays what is missing in ELK log collection?