I'm looking for advices for a new cluster. I would like to manage my firewalls logs with an ES cluster.
I have five sites in five separate cities.
So, my idea is to install an ELK stack on each site in order to collect the syslog logs from the local firewalls. By doing this, I'm sure that logs are collected even if a link between two sites is broken.
These five stacks will be my five nodes. So every node will be a "data node".
Originally my idea was to create an index every day. But after thinking, I think there will be too many shards in my cluster. For example, for one day and if I keep the default settings, there will be 5 indices, each divided in 5 primary shards and 5 replica shards. So 5*5 = 25 primaries shards and 25 more replicas shards ==> 50 shards per day.
What do you think of it ?
In your point of you, what is the best in terms of reliability : create a unique index ? or create daily indices ?
In fact, I would like to used Curator for deleting the indices older than 7 days. Is it possible to delete documents older than 7 days ? I didn't find anything.
Moreover, what is for you the best number of shards ? or how can I calculate it. Is there a ratio between number of nodes and number of shards ?
Thank in advance,
I'm waiting for your precious advices.