We need to deploy elastic stack in production to consume syslog data (amount of data is moderate)
During POC we deployed all ELK component in a single VM. In production how should we deploy ELK stack to ensure our environment is resilient.
Elastic search - 3 VM (One master node, two data node) or I could have all three in single VM, do we actually need to create separate node?
2)One VM each for Kibana & Logstash?
How resiliency can be achieved if my primary instance goes down, should we have separate instance of each component to achieve resiliency?
Is there any component in elastic stack which takes care of resiliency like zookeeper ?
Yes, you should create different VMs for different ES node. In case you create all ES on one VM and VM goes down then all ES goes down. However, elastic community recommend running 3 dedicated master nodes per cluster having dedicated master nodes which run in their own JVM increases stability and resilience as they are not affected by garbage collection that can affect other types of nodes. These nodes do not handle requests and do not hold any data, and therefore only require less resources (such as CPU, RAM and Disk).
Yes, VMs shoud be seprate for both components.
You can achieve the resiliency using recommended cluster and Hot-Warm architecture in elasticsearch as there is no components like zookeeper in elasticsearch.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.