Secure submission of filebeat from remote

Elastic Stack Logstash
1

Hi!

I would like to set up filebeat and logstash for multiple networks (~100).
As we are four admins who need the logs, we only plan a small deployment (1x elasticsearch, 1x logstash and 1x kibana - nodes).

This deployment has it's own private net and can be made reachable from the outside using port forwarding.

I would like to have all hardware (servers, vms, routers, switches, etc.) send their logs to this central installation.

Is this possible? The startup guide does not explain authentication (at least I did not find it). Also, many VMs are not trusted: They are rent by others who are able to browse all files located on it.

How is logstash / beats deployed on heterogeneous infrastructures?

2

How much data do you envisage coming from all these machine data endpoints?
Daily in MB or GB? & Weekly? & Monthly?
What will your retention period be?

Heterogeneous:
For windows servers there is winlogbeat -> LS
For linux filebeat -> LS

How many end points will you run *beats on?
These *beats will they all send logs to a single LS service?

Many *beats -> LS -> ES -> Kibana.

How much resources will you provision to LS, ES and Kibana?

3

The amount of data is currently unknown in total but we plan about 100MB per server daily.
About 400 VMs will send data to us (30% windows, 40% linux, 30% embedded like router, etc.).

Currently, we would like to keep the last 30 days saved. Older logs are saved to backups, the production system is only for debugging.

Ressources for the LS-stack are flexible. For the start we got 2 CPU (24C) and 144GB RAM with local RAID10, 6TB (spinning SATA) and writeback cache. We are planning to use KVM or Proxmox on top.

I found a reference for users in the logstash docs:
https://www.elastic.co/guide/en/logstash/current/ls-security.html

Using this guide, I will try to deploy Auth + TLS.
To add redundancy for beats, I am planning to use multiple DNS-A-Records, anything wrong with this?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.