Hello. Could anyone please help me to deside how to scale elastic to a multi-server cluster.
Current situation:
1 linux machine 2 cores/8GB RAM
1 elastic instance + 1 kibana instance
40 data indexes every month, 1 shard/0 replicas for each
Number of new indexes/month grows slowly, consider 5 indexes per year
Average size of index 3.2 gb with total monthly data of 125 GB with 25GB growth per year
210 Million events per month with average event size of 610bytes
What we do is mainly just index all those events while doing some non-aggregating queries from kibana each 5 second (for example we often do "catch OR failure OR error OR rollback OR warning" query).
At current event rate we have a problem with query execution time - it now sometimes exceedes 5 seconds or even more, so we decided to consider scaling horizontally.
I've read a lot of articles about elastic sizing, all of them only claim that "It depends" which is not useful at all.
I also need some architectual insight here, as I can not quite decide how much nodes do we need. For instance, we could have 3 nodes, 2 of which can be master+data nodes, and one will host kibana and serve as master+non-data node. Is there any sence in such configuration?
So overall question is what is the best architecture decision in this situation and what could be advised for sizing for that arch.
Thanks to anyone who responds.
Dmitry