Architecture Desing CCR

Elastic Stack Elasticsearch
1

Dear Team,

We are planning to implement an on-premises Elastic SIEM/EDR solution for an OT/IT security environment comprising approximately 60 endpoints and an estimated daily log ingestion volume of 50 GB/day.

Before finalizing the High-Level Design (HLD) and proceeding with implementation, we request your technical validation of the proposed architecture and VM resource allocation.

1. Proposed Architecture and VM Sizing

Site VM Proposed Components vCPU RAM
DC VM-1 Vector + Kafka + Logstash (Ingestion Gateway) 4 12 GB
DC VM-2 Elasticsearch (Hot/Master Role) + Kibana 8 15 GB
DC VM-3 Elasticsearch (Cold Role) 4 8 GB
DR VM-4 Vector + Kafka + Logstash (Ingestion Gateway) 4 12 GB
DR VM-5 Elasticsearch (Hot/Master Role) + Kibana 8 15 GB
DR VM-6 Elasticsearch (Cold Role) 4 8 GB
Third/Neutral Site VM-7 Elasticsearch (Frozen/Master Role) and object storage connectivity 4 8 GB

Object storage: Approximately 10.1 TB proposed for archive/frozen data.

2. Validation Required

Kindly review and confirm the following:

  1. Ingestion capacity: Are the proposed CPU and RAM resources sufficient to process 50 GB/day of OT/IT security logs, including peak ingestion rates and burst traffic?

  2. Ingestion gateway sizing: Is it appropriate to deploy Vector, Kafka, and Logstash together on a single VM at each site with 4 vCPU and 12 GB RAM? Please validate Kafka throughput, queue retention, buffering, and back-pressure handling.

  3. Elasticsearch sizing: Are the proposed resources for the Hot and Cold nodes sufficient for the expected ingestion volume, search performance, security analytics, alerting, dashboards, and concurrent SOC users?

  4. Master-node architecture: Is the proposed combination of Elasticsearch Hot and Master roles on VM-2 and VM-5 recommended for production? Please advise on dedicated master nodes and the minimum supported topology for this deployment.

  5. DC–DR architecture: Is the proposed architecture suitable for disaster recovery? Please clarify the recommended approach for cross-site replication, cluster separation, failover, data consistency, and recovery objectives (RPO/RTO).

  6. Third/Neutral Site: Please validate the proposed Frozen/Master role on VM-7. Should the third-site VM act as a voting-only master/quorum node, a frozen-tier data node, or separate components? Please clarify the supported design and whether the proposed 4 vCPU and 8 GB RAM are sufficient.

  7. Retention and storage: Please validate whether the proposed 10.1 TB object storage is adequate for the required retention period, considering 50 GB/day ingestion, Elasticsearch indexing overhead, compression, replicas, snapshots, and growth.

  8. OT/IT security requirements: Please confirm whether the proposed sizing supports security monitoring and detection use cases for network devices, firewalls, servers, endpoints, and OT/ICS-related data sources.

  9. EDR requirements: Please clarify whether the proposed sizing includes endpoint telemetry processing, EDR detection, and associated integrations, or whether additional Elastic Security/EDR components and resources are required.

  10. Final sizing recommendation: Please provide the OEM-recommended production sizing for each VM, including vCPU, RAM, disk capacity, storage performance/IOPS, and network requirements, along with any architectural changes required.

3. Confirmation Requested

Please confirm whether the proposed architecture and VM resources can be approved for production implementation at the stated ingestion volume, or provide a revised, OEM-supported architecture and sizing recommendation..

Regards

Chandra kamal

2

please Team Reply on this post , this is urgent

3

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

4

This forum is a community forum, there is no SLA or urgency here, you need to wait to see if someone can help you or provide some feedback since everyone here is a volunteer that contributes when they have time.

But, as mentioned in your previous question, your number of 50GB does not make much sense, it is not clear how you reached to this number, this is too small, if this is your daily ingestion your infrastructure is overkill.

Also, you mention using object storage and frozen tier, but those requires a paid license, it is not clear if you are planning to buy a license or already have one, if this is a requirement you need to get in touch with Elastic and they can help you plan your cluster better.

Being honest, the best approach is to do a proof of concept with a small cluster to get real data usage, you can use a 3-node cluster for this, when you have what would be your real data usage you can estimate better.

If you provide more context about your requirements people would be able to help you better, but based on what you provided until now, all those resources are overkill and will be idles/unused for most of the time.

5

I just want to understand the use case for this cluster. It looks like you are trying to serve two different "sites".

Are each of these sites expecting to ingest 50 GB/day? so 100 per day?

What is the use case for 2 seperate Kibana instances? Redundancy?

As you work to mature these clusters or cluster to a Prod stat, You will want to stand up a seperate monitoring cluster to ship out all the internal monitoring logs. In small clusters like these the monitoring ingest alone can cause quite a bit of unwanted ingest, especially on innitiall set up while you have some "noisey" beats.

6

Thanks for the feedback. Let me provide the complete project context, because my main question is not only whether 50–60 GB/day is small, but what a practical production architecture and sizing would look like for this requirement.

This is an on-premises OT/ICS SOC deployment across two physically separate sites (DC and DR).

The requirement is to run two independent Elasticsearch clusters, one in DC and one in DR. We are not planning to use CCR, because we want both sites to maintain their own searchable copy of the event data.

The intended data flow is:

Sources → Ingestion → Kafka → duplicate/replicate stream → Logstash → Elasticsearch-DC

and the same logical event stream is also delivered to:

Kafka/replication → Logstash → Elasticsearch-DR

Therefore, 50–60 GB/day is the estimated logical source data, not 50–60 GB/day of unique data per site. Both Elasticsearch clusters would index approximately the same volume, so the aggregate physical ingestion across both clusters would be approximately 100–120 GB/day.

The project scope includes:

  • Elastic SIEM
  • Elastic Defend / EDR
  • Elastic Agent + Fleet
  • OT/ICS monitoring
  • Zeek for passive network monitoring
  • Kafka + Logstash ingestion
  • DC/DR data duplication
  • 1-year retention
  • 10 days Hot
  • 20 days Cold
  • 335 days Frozen/Searchable Snapshot

We are currently trying to determine the right production sizing, rather than intentionally deploying a large infrastructure.

What I need guidance on is the following:

1. Elasticsearch cluster sizing

For approximately 50–60 GB/day logical ingestion and 365-day retention, with the same data maintained in two independent sites, would you recommend a 3-node Elasticsearch cluster per site as the starting production architecture?

If yes, what would be a reasonable specification per node in terms of:

  • vCPU
  • RAM
  • SSD/NVMe capacity
  • HDD capacity for Cold
  • Storage IOPS / performance considerations
  • Replica configuration

2. Hot / Cold / Frozen architecture

For this ingestion rate, should Hot, Cold and Frozen be deployed on:

  • separate dedicated nodes,
  • or can these roles be combined in a small 3-node cluster?

In particular, would you consider a design such as:

3 Elasticsearch nodes/site with Hot + Master roles

and using searchable snapshots/object storage for older data,

or would dedicated Cold/Frozen nodes still be justified?

3. VM count

I specifically need guidance on a practical VM count per site.

For example, can components such as:

  • Logstash
  • Fleet Server
  • Kafka
  • Kibana
  • Elasticsearch

be safely co-located for this workload, or should they be kept on separate VMs?

I am looking for something that is production-defensible but not unnecessarily oversized.

4. Kafka architecture

Since Kafka is being used as the duplication/recovery boundary, should I deploy 3 Kafka brokers per site even at 50–60 GB/day, or would a smaller Kafka deployment be reasonable?

Also, for DC-to-DR duplication, would you recommend MirrorMaker 2 for this use case?

5. Logstash sizing

What would be a reasonable number of Logstash instances and VM sizing for approximately 50–60 GB/day, including possible ingestion peaks?

Would 2 Logstash instances per site be sufficient for HA, or is a larger pool normally required?

6. Kibana

The reason for having Kibana in both sites is site-level availability. The intention is to expose both through an HA LB/VIP with a single SOC URL.

Is 1 Kibana instance per site reasonable for this workload, or would you recommend 2 Kibana instances per site?

7. Fleet / Elastic Agent

Elastic Agents will be installed on the actual Windows/Linux endpoints and managed through Fleet.

For the server-side management plane, what would be a reasonable Fleet Server count and VM sizing per site for this type of deployment?

8. Zeek

Zeek will run as an independent passive network sensor and send its logs into the Elastic ingestion pipeline.

Should Zeek be sized completely independently from the Elasticsearch cluster, and are there any specific considerations you would use when estimating its VM resources?

9. DR / Active-Active question

The intended model is two independent Elasticsearch clusters with continuously duplicated event data, rather than CCR.

Is this architecture reasonable for a 50–60 GB/day workload, or would you recommend a different design for DC/DR?

10. Monitoring

You mentioned that a separate monitoring cluster may eventually be useful.

At this scale, would you recommend:

  • monitoring the production cluster initially using the same deployment,
  • or deploying a separate monitoring cluster from day one?

11. Frozen / Searchable Snapshots

The requirement currently includes 335 days of Frozen/Searchable Snapshot data.

Assuming the appropriate Elastic subscription is available, what would be the recommended storage architecture for this — and should the object-storage sizing be calculated independently for each site?

12. Final question

Most importantly, if you were designing this project yourselves, what would you consider a reasonable minimum production architecture per site for:

50–60 GB/day, 365-day retention, SIEM + EDR + OT monitoring, Kafka-based DC/DR duplication, and HA?

I would particularly appreciate a practical example such as:

“X VMs per site, VM-1 = __ vCPU / __ GB RAM / __ GB storage, VM-2 = __ ..., Elasticsearch = __ nodes, Kafka = __ brokers, Logstash = __ instances, Kibana = __ instances.”

The objective is to avoid both under-sizing and unnecessary over-sizing, and to arrive at a production architecture that can be justified technically.

7

Yeah, this looks like a simple question, but in the end is not that simple, to avoid under and over provisioning you would need better estimates about how much data you have daily and what would be your expected event per seconds rate.

To get those numbers people normally do a proof of concept cluster, a small one, a single node would work, with just a couple of servers configured to get real data usage and then estimate based on the real data numbers.

elasticsearch

But assuming that you will have 60 GB/day, if we include replicas you would have 120 GB/day per site.

So this would be around 3.6 TB per month and something close 43 TB year

For 10 days in hot, you would need something close to 1.2 TB, considering that you need to have extra space for internal indices as well, I think you could use 2 hot nodes each one with 1 TB nvme disks.

I don't think a cold tier is justified here, you could just use hot-frozen, for the frozen tier you could use 2 frozen nodes with 1 TB HDD disk, which is used for the cache for the searchable snapshots and then you would need a NFS share around 40 TB, this can also be HDD, no need to use SSD.

If you want dedicated nodes for every role this would be a 7-node cluster, with 3 dedicated master nodes, 2 hot nodes and 2 frozen nodes, the specs depends a lot on the use case, just start small and increase the memory/cpu if needed, maybe 2 GB RAM for the master and frozen nodes and 4 GB RAM for the hot nodes, you can also check the profile and ratios that elastic uses in their cloud here.

But being honest, for this kind of workload a 3-node cluster with all roles may be enough, just increase the disk size.

For monitoring is always recommended to have a separate cluster, this can be a single-node cluster.

kibana/fleet

Regarding the other tools, unless you want to have redundancy for everything I do not think you need 2 Kibanas and 2 Fleets from the start, so 1 Kibana and 1 Fleet would be enough, I also do not think that you should have a Fleet in the DR site, Elastic Agents can be enrolled to one single cluster at time, a Fleet in the DR means that you would need to re-enroll your agents to be able to manage them there.

logstash/kafka

This is also something that depends on the event rate of the ingestion, but 60 GB/day is pretty small, having logstash in HA here may be unnecessary, but if you want you could use 2 logstash nodes acting as both producer and consumer, I see no reason to have 2 logstash as producers and 2 logstas has consumers, the data is too small.

You could just use a single-broker kafka cluster as well, if you want redundancy you would need a 3-broker cluster.

Ideally you would have 7 machines here, 2 logstash as producers, 3 kafka brokers in a cluster and 2 logstash as consumers, but this can work with just 2 machines.

I have both scenarios in production, in some cases I have 7 machines and in other just 2.

I do not have a DR, so cannot provide any feedback about mirrormaker or replication to a different site.

I do not zeek as well, so no idea how you do here.

Besides these recomendations I strongly recommend that you do a proof of concept cluster to validate this, spin-up small clusters, even single-node ones, use the trial license to validate the enterprise features.

Always start small and increase as needed.