Hello all. Great conference by the way. Excellent question answer sessions and breakout talks. One question I wanted to ask is that I have setup an ELK stack to monitor 1000+ nodes with filebeat, metricbeat, Suricata and Packetbeat. In production we have 5 Elasticsearch boxes with Logstash consuming beats and everything works great.
But, we need a test cluster. It would be very beneficial to have a cluster where I could test upcoming logstash configuration changes which entail bouncing the instance, making template changes, hot/cold storage configs, etc.
What is a good approach to building a low cost test cluster? Might sound nuts, but could you take something like K8's and then spin up 1000 or so VM's and an ELK Cluster? Would a certain fraction of the VM's be sufficient for testing... Just an overall question of what test cluster footprints look like and typical deploy strategies between them?