Topology:
3 Master nodes
2 Client nodes (for data intake)
2 Client nodes (for querying)
4 Data nodes (now 8, we added 4 after the crash)
All machines 4cpu's and 15gb memory.
1612 indices with 10 shards each and 1 replica
Total data size 662GB
Daily Intake ~ 40gb
Elastalert runs about 30 varying queries every 3 minutes against dedicated client nodes.
Devs run queries and aggregations against the cluster regularly via kibana.
I guess I'll follow the rolling restarts instructions for the maximum file limit setting.