Hi there,
I've just finished a POC piece to show the benefits of using the ELK stack to aggregate logs from our applications.
Now that we have some traction for the project i need to design this with a global mindset and I have one main question for now.. One big cluster using index names to separate regions or to go for smaller localised clusters with a tribe node to enable querying across the geographical DCs.
bit of background -
3 Global DCs North America, Australia and EMEA.
2 are AWS and the other with a co-location/hosting partner
1 DC produces approx 10GB GB logs per day the other 2 are significantly smaller but will grow in time.
We are aiming to start small (with the largest DC) and add services in gradually and expand the cluster as needed to avoid over provisioning.
Any opinions/advice would be appreciated.
Cheers.