Hello everybody,
I am currently taking over our Elastic cluster at my company after the person who set it up and managed it so far left the company. Given my limited experience with the ELK stack, I have a few questions when it comes to setting up the environment appropriately and choosing the correct size.
Currently, we have about 470GB of data on our Elasticsearch nodes. The way it is set up is as follows: 1 Ubuntu virtualized host system with 8 CPU cores, 32GB of RAM, and 2TB SSD disk space. The Ubuntu machine hosts Kibana and two Elasticsearch containers with a replica of 1. Right now, the whole environment is running on version 7.12 and was built using the ElastDocker GitHub repository.
While digging through the documentation for a bit, I was confused as to why my predecessor set up the environment this way, since there is no failover in case the one Ubuntu VM crashes.
I am now wondering the following: What size should our cluster be, in terms of nodes with which computing resources? I know this depends on a few factors and will be specific on a case-by-case basis, but with 470GB of data already and about 3GB of additional data per day, which should be accessible for 2 years, and the following Elastic metrics. Maybe someone can give me a rough guideline on how I should go about structuring it.
Running 7.12 also seems like an ancient old version, so how easily can I upgrade to the latest 8.x version? Can I, in theory, add a third container to our cluster that is already running the higher version and do a rolling update, one node after the other, without having to take the whole environment down?
Also are there sources to learn about setting up the whole Elastic infrastructure, such as courses, videos, or tutorials, you guys can recommend for a beginner? I noticed that there is the Elastic-Engineer certification itself, but that seems more targeted at developers using the data than for admins setting up the environment, right?
Any advice and pointing in the right direction is much appreciated. Cheers!
