First Claster for log collection

[zenekas1978 (Zenon)]

Hi All,

I would like to build my first ELK cluster in Azure in order to collect and analyze logs from VM's and firewalls. I used ELK Azure template and I created 3 masters, 1 data, Kibana and Logstash nodes.

I would like to monitor ~100 VM's and ~20 network devices.
I plan to keep logs 30 days.
Most of VM uses Windows Server so I plan to use winlogbeat to send logs to ELK.

How many nodes I should use for cluster?
It's very hard to calculate storage size I will need.
Do you think the above scenario is enough? or I should use a different schema?
I used azure Ds1v2 for all nodes. (Data + 512 GB drive ssd)

Thanks for help and advice.

[warkolm (Mark Walkom)]

Welcome to our community!

I think you will probably run out of space with that many VMs. You might want to setup your cluster, ingest some data and then see what it can hold and extrapolate from there.

[zenekas1978 (Zenon)]

Hi Mark,

Do you have any proposition for me?
How my cluster should look ? Do i need 3 masters and 1 data node or i should use different schema?

[Christian_Dahlqvist]

The best cluster configuration to start with is generally 3 data nodes that also are master eligible. Once the cluster grows you can start exploring dedicated node types.

[zenekas1978 (Zenon)]

Thanks Christinan for the advice. I will try to customize Azure ELK template so I will be able to setup 3 nodes with master/data type + Kibana + Logstash. I connected one server during the last 5 days and it used 980 MB from my space.