3 server design recomendations

Elastic Stack Elasticsearch
1

all,

any input on previous experience would be greatly appreciated. I've got access to 3 Dell R610's with 6 (74Gig 15k SAS drives). I'm not certain on the RAM yet but most systems were between 24-32Gig.

the main function of this ELK install will be to take in syslog data from about 150 or so network devices (routers, switches, wireless controllers, firewalls ect). Our production environment takes about 400-500meg of data per day...

curl http://localhost:9200/_cat/indices?v

that command shows on average 400-500meg per index which is created daily from what I can see on our test system.... Assuming this new system has no performance issues we will likely turn up our logging level to expect to see between 2-4gig of data per day.

Questions...

  1. Should I install all 3 systems with Raid0 for fast I/O?
  2. Should I make all 3 nodes elastic nodes and then just run kibana/logstash on 1 of the 3 nodes?

Each server will have 4 10gig NIC's on them (not that I need 10g) and what I'm thinking is creating a back end network for elasticsearch that's on a non routable VLAN to help secure the system and to offload any network activity between the three nodes. so basically all 3 servers will be dual connected... they'll have a backend non routed 10G network for elasticsearch connectivity and they'll all have a front end 10g connection for basic system management. I'll also need to use this front end connection when connecting to kibana and for syslog's to get into the system.

any suggestions are greatly appreciated.

this is also a fresh install. no need to migrate any data from an existing ELK instance.

thanks,

Lee Carter

2

Yes. Use replicas to give you redundancy.

Yep!

3

thanks. any major concerns you see if the servers are actually set using Raid5? I'm not a systems guy but our server admin is recommending raid5 to save from having to rebuild a system in the event of a single drive failure. thoughts or concerns on that?

4

It's sloooooow.
Also - Google

5

thanks. I'll take that back to them. perhaps if they are concerned we can do raid 1 on two disks for OS redundancy then make the data volume raid0 or something like that for speed since all data is redundant across clusters. I think their concern was having to rebuild a server in the event of a drive failure.

6

Yep, good idea.