Is ELK for me?

Hi,

First of all I would like to apologize if my story is a bit vague but that's because most of it is still in a planning stage and I'm still looking how I can best achieve what I want.

My situation is as follows;

I have a couple of hundred (might grow to a thousand) remote locations that I need to monitor. Specifically data generated by pfsense (data usage, firewall logging etc) and maybe server status. But monitoring the firewall is by far the most important.

I haven't installed ELK yet but google tells me I can get pfsense data into ELK and have it displayed in a dashboard.

Question:

  • Can I have a separate dashboard for each location?
  • Can I also create groups to group various locations together?
  • I suppose data can be generated over a certain time frame? E.g. default view is 24 hours but if required you can select a custom range?

I would also like to know about alerting and liming access. It looks likes X-pack is what I want for this. I need to give access to techs but I also want to give access to customers so they can view certain data related to their locations (e.g. a custom customer dashboard).

I also need report. For example if a location has not sent any data for x amount of time or data usage is more than x amount over x amount of time sent a alert email and show it on a "alert dashboard".

Question:

  • Will X-pack allow me to have different dashboards, access limitations etc for techs and customers?
  • Can I get automated alerts for each location based on rules I set?

The final question related to on site or cloud hosting. As much as anything this will be a matter of costs. I don't expect each location to generate a lot of data (couple of MB per day at most) but we do need to store logs for at least 6 months. 1 to 3 years would be preferred.

I know there is not one size fit all answer for this but how much time will we be spending on maintaining ELK if we had to process give or take 1GB of data a day?

Yes

Yes

Yes

(There are current limitations here, but) Yes

Yes

Not much once it's up and running. You could also use Elastic Cloud and not worry about it :slight_smile:

Thanks for the reply.

I've installed ELK and it looks like its going to work, just need to transfer to a pc with more ram. The 1.5GB ram available to the VM on my laptop isn't going to cut it hehe.

Just one question with regards to different locations, how should I go about this? I haven't found any documentation on this (yet).

We have a lot remote pfsense boxes with dynamic IPs. I've seen you can define client IPs in the config files but as we'll have to work with dynamic IPs that isn't going to work.

Is there any way to handle this inside ELK by maybe looking at something like pfsense hostnames (tbh I haven't really looked at whats inside the pfsense logs yet) or do some pre-processing before sending out the logs (Beats?). Or is the only solution to do something like running a VPN between the remote client and the server so each location has a known IP?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.