First of all I would like to apologize if my story is a bit vague but that's because most of it is still in a planning stage and I'm still looking how I can best achieve what I want.
My situation is as follows;
I have a couple of hundred (might grow to a thousand) remote locations that I need to monitor. Specifically data generated by pfsense (data usage, firewall logging etc) and maybe server status. But monitoring the firewall is by far the most important.
I haven't installed ELK yet but google tells me I can get pfsense data into ELK and have it displayed in a dashboard.
- Can I have a separate dashboard for each location?
- Can I also create groups to group various locations together?
- I suppose data can be generated over a certain time frame? E.g. default view is 24 hours but if required you can select a custom range?
I would also like to know about alerting and liming access. It looks likes X-pack is what I want for this. I need to give access to techs but I also want to give access to customers so they can view certain data related to their locations (e.g. a custom customer dashboard).
I also need report. For example if a location has not sent any data for x amount of time or data usage is more than x amount over x amount of time sent a alert email and show it on a "alert dashboard".
- Will X-pack allow me to have different dashboards, access limitations etc for techs and customers?
- Can I get automated alerts for each location based on rules I set?
The final question related to on site or cloud hosting. As much as anything this will be a matter of costs. I don't expect each location to generate a lot of data (couple of MB per day at most) but we do need to store logs for at least 6 months. 1 to 3 years would be preferred.
I know there is not one size fit all answer for this but how much time will we be spending on maintaining ELK if we had to process give or take 1GB of data a day?