Hi Team,
I am new to ELK and tried to read as much as I could. I came to the conclusion that I would want to implement a hot warm architecture for my setup. Unfortunately my client has provided me with limited VMs and CPU so I am restricted.
Currently we have v7.3.2 of Elasticsearch. Indexes in a day account about 500MB only.
I am thinking to make use of 2 servers.
Server 1 - 4 vcpu, 8GB RAM and 20 GB local storage.
Server 2 - 2 vcpu, 2GB RAM, 20 GB local storage and 1 TB NFS storage.
I intend to apply the hot warm architecture in this manner -
- Server 1 will be my hot node.
- Server 2 will be my warm node.
- I will apply a index lifecycle to move indexes greater than 30 days or if the space is 85% full then move the indexes to warm node.
- On the warm node, utilize both the local and NFS storage to save indexes. I read that the indexes will be saved to the NFS when the local storage is full.
- Delete indexes beyond 12 months.
I heard the retreival from NFS storage is slow but I guess we can live with it since we are concerned about the speed with only the recent indexes.
Also since logstash points to the cluster and not the node. Given that we have only 2 servers, where and how many logstash nodes should I configure ?
As mentioned my constraints, could you guide me if the above seems about right?
The current architecture hasn't been implemented yet and I can implement the architecture based on your recommendations.