I got a monster server from my management to build a ELK stack. They are not after highly available cluster. They are good with one server too. The following is the specs:
Processor: Intel XEON 2.2Ghz (2 Processor)
Memory: 32GB (soon it will be 128G)
System type: 64 bit
OS: Windows Server 2012 R2 (yes we can't build Linux based ELK)
Disk: Well, disk configuration is interesting and overkill but this is what I got. Here is the complex configuration:
There are total 72 SSDs 800GB each in this server. There are three RAID controllers that are configured as RAID 1. Each RAID controllers are assigned 24 SSDs.
In Windows, 9 volumes are stripped in to 3, 17TB each using dynamic disks. So, currently I can see 3 volumes of 17TB and C: drive of 700GB.
Now, we have a plan to dump all of all Windows security events regarding authentication and file auditing, Microsoft Exchange messaging logs, SharePoint, and IIS logs to this server.
Should I install Logstash on same server or any virtual server. I believe, I should install Kibana and Elasticseach on this server. How should I configure Elasticsearch as above-mentioned scenario?
We want to keep data for at least 90 days. Currently, we are dumping Windows security events that includes authentication only and Exchange messaging logs that generates around 100 events per sec. Once we start logging File auditing too, it will jump to 300-400/sec I guess.
Okay, so for a 90-day retension you might be looking at ~1 TB data but that obviously depends on the size of the events. That should be doable for your server.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.