Hello and welcome,
Is this estimate of 50 GB/day correct? This is very small and your configuration is extremelly overkill for it.
How did you arrive to this number?
The overall design is ok, I have a similar one specially the logstash + kafka + logstash, but some things are a little confusing.
First, your logstash layer that act as producers do not exist in your DR, so your DR would be only for querying the existing data, right?
Another thing, Fleet is used to manage Elastic Agents, and Elastic Agents can be enrolled to one cluster only, there is no reason to have DR fleet servers unless every time you change to DR you reenroll all your agents into the DR cluster.
Also, as far as I know MinIO is deprecated, what would be the use here? Local object storage compatible with s3 api to use on frozen tiers? I'm not sure what is being used on-premises now, but since this requires a paid license it may be better to reach to elastic to have some help designing your cluster.
this arch recommended by OEM for multisite having DR and below is the sizing details also recommende by OEM
| Aspect | ||||||||
|---|---|---|---|---|---|---|---|---|
| Data Ingestion per day | 50 GB | |||||||
| Retention Period | 1 year - 10 days hot | 20 days cold | 335 days frozen | |||||||
| Deployement | Self managed | |||||||
| Total RAM (GB) | Total Storage (TB) | Object Storage(MinIO) (TB) | ||||||
| Hot | 30 | 0.8 | ||||||
| Cold | 16 | 1.2 | ||||||
| Frozen | 8 | 0.6 | 10.1 | |||||
| Kibana | 8 | |||||||
| Total RAM | 62 | |||||||
| BOQ | ||||||||
| Data Tier | Nodes | Total CPU | Total RAM | Total Storage(TB) | Object Storage(TB) | Type of Disk | ||
| Hot | 2 | 16 | 30 | 0.8 | SSD/NVME | |||
| Cold | 2 | 8 | 16 | 1.2 | Dense HDD | |||
| Frozen | 1 | 4 | 8 | 0.6 | 10.1 | Object Storage | ||
| Non Data Nodes | nodes | Total CPU | Total RAM | Storage(GB) | Type of Disk | |||
| Kibana nodes | 2 | 4 | 8 | 300 | SSD/NVME now let me know how shall i create architecture for customer having 60 endpoints from where we will take data and this is OT/IT SOC setup in which they have below telemetry |
Requirement
Detail
Current SIEM ingestion
Approximately 50 GB/day
EDR scope
35 workstations + 25 servers, including 4 Linux servers
Current SIEM source
Netka Syslog