We are planning to deploy elastic stack for logging and monitoring as SIEM, we want to start from open source version (community version) and if we see value we would upgrade to enterprise version with full security features and support. We have multiregional on premise and cloud presence and wanted to review attach design.
We want to deploy multiple ELK clusters in different region close to log sources to avoid WAN link consumption and cloud egress charges
We still want all clusters to be able to integrate with our master ELK cluster in on premise DC and we can see and query logs from all cluster across different regions and cloud.
We understand it could be bit slow when querying or searching log of different region but that is fine for us.
We want to archive more than 90 days logs automatically to object storage in cloud to reduce storage cost but still be able to search those logs from Kibana in DC on need basis, speed is not a problem. we want to keep only 1 year log in cloud object storage.
Not sure if those two requirements work well together, while you can have a central cluster doing Cross-Cluster Search on remote clusters, you will still have egress data because the requests and responese for queries will still need to be sent to the Central Cluster.
Also, you want to have a cluster per region or per cloud provider? If you want a cluster per region, are you talking about 10 different ELK clusters + 1 Central Cluster? This would need a lot of work to manage and you will probably spend more on the resources for those cluster than you would spend on data egress.
A better approach, and what is more common to see, would be to have a couple of machines on each data center running Logstash to receive the logs and send them to your ELK Cluster.
This is what Elastic calls frozen tier, it uses searchable snapshots, but this is not possible with the basic license, you can only use searchable snapshots with an Enterprise license.
You can test this for 30 days if you enable the trial license.
Thank you for looking into, I think we can consoildate in region on single cloud and make it more simpler.
We are fine with enterprise license as long as we have capabilties to store 90 days logs in ELK stack as SIEM and archive after 90 days to S3 bucket for 1 year. And we can search through archived logs on demand (slow is fine ).
Can logstash compress logs to send before destination.?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
