We are looking to setup ELK stack on Google Cloud Platform and would need some suggestions to build a high availability cluster.
We have disabled Stackdriver Alerts for Dev/QA Environments and are looking to use ELK as its an Open Source Alternative for Log Monitoring and Analysis. We also have Kubernetes Clusters, through which we get our application logs.
Would you recommend to build a ELK cluster on GKE and run ELK components as Pods or do I set it up as Virtual Machines ?
Also, as there is no filebeat client involved, how do I pull logs from GCP ?
We have created exports ( sinks ) in Google Cloud Platform which can be used as Pub/Sub. But, we are not sure how to setup a connection between GCP Logs and Push/Pull Logs to Logstash/Elasticsearch