Hello, we are developing a searching engine with elasticsearch and already have 3 nodes on dedicated servers for each.
We almost have read the documentation, but faced to a couple of problems.
about our servers:
we have 3 servers with the following characteristics: 64 RAM, 16 CPU, 2000 GB nvme.
we have two main indices. that WILL contain(in the future) billion of documents(now service not started yet). so, in three nodes, we have 3 primary shards and 2 replicas for each index. (each node will contain one primary and one replica)
design of nodes: nodes — ImgBB
So, my questions is:
as written on the documentation, each index contains pieces of segments(in our case, one index contains 3 primaries, 2 replica shards).
but, on docs, you recommended to use 20GB - 40 GB memory for each shard.
if each of our shards will contain 30GB of memory, then, 3 primaries will take 90GB + 2 replica (60),
total, we will use for two indices, 300GB of memory.
the first question is:
can we add primary 4 to one of the existing node. (for example to the node-1), then store its replica to another node.
and, if we will add one more shard, then will we add to the existing nodes, (If it's possible, I would like to add 3 primary and 2 replicas shard for each node)
or, if I'm wrong, can you suggest me, what should we do, and how we should allocate our shards between three nodes that we can easily expand in the future.
we can not change number of primary shards, after creating the index, but can change number of replicas. so, if we want to expand, should we increment number of replicas only, or we should reindex?.
I have also a few questions, but it depends on the answers to the questions above.