New ELK Setup: Hardware and Disk Sizing?

Elastic Stack Elasticsearch
1

Hi,

I got a monster server from my management to build a ELK stack. They are not after highly available cluster. They are good with one server too. The following is the specs:

Processor: Intel XEON 2.2Ghz (2 Processor)
Memory: 32GB (soon it will be 128G)
System type: 64 bit
OS: Windows Server 2012 R2 (yes we can't build Linux based ELK)

Disk: Well, disk configuration is interesting and overkill but this is what I got. Here is the complex configuration:

There are total 72 SSDs 800GB each in this server. There are three RAID controllers that are configured as RAID 1. Each RAID controllers are assigned 24 SSDs.

In Windows, 9 volumes are stripped in to 3, 17TB each using dynamic disks. So, currently I can see 3 volumes of 17TB and C: drive of 700GB.

Now, we have a plan to dump all of all Windows security events regarding authentication and file auditing, Microsoft Exchange messaging logs, SharePoint, and IIS logs to this server.

Should I install Logstash on same server or any virtual server. I believe, I should install Kibana and Elasticseach on this server. How should I configure Elasticsearch as above-mentioned scenario?

2

How many events per second? How long will you keep data around?

3

We want to keep data for at least 90 days. Currently, we are dumping Windows security events that includes authentication only and Exchange messaging logs that generates around 100 events per sec. Once we start logging File auditing too, it will jump to 300-400/sec I guess.

4

Okay, so for a 90-day retension you might be looking at ~1 TB data but that obviously depends on the size of the events. That should be doable for your server.

5

Can I install Logstash on same server or any virtual machine will work too?

6

Second, as I have mentioned, I have three volumes of 17TB. Can I use multiple data path with Elasticsearch?

7

Can I install Logstash on same server or any virtual machine will work too?

Either way. For just a few hundred events per second you don't need anything powerful.

Second, as I have mentioned, I have three volumes of 17TB. Can I use multiple data path with Elasticsearch?

Yes.

8

Hi Magnus,

Thank you for your reply but how can I use those three volumes in config file?

9

Thank you for your reply but how can I use those three volumes in config file?

There are examples of multiple path.data values here: Directory layout | Elasticsearch Guide [8.11] | Elastic