I'm trying to ingest 1.5 PB of data into elastic search. My question is, if I ingest data directly through co-ordinate nodes instead of using data nodes, what will be the consequences and how it will affect the performance?
I have cluster of 50 data nodes, 3 master nodes and 2 co-ordinate nodes(out of 2, Kibana is installed in one of the co-ordinate node).
the coordinating nodes are responsible to forward the requests to the corresponding data nodes containing the respective shards where the data will be indexed.
If you have two coordinating nodes, those will be responsible forwarding all of your data to 50 data nodes. I cannot tell if this is a theoretical or practical bottleneck, but just take the data into amount, that needs to be transferred from those two nodes to the other 50. Maybe it makes more sense to send this to all the data nodes, so that the load is spread more even (given the fact you are indexing into all 50 nodes, something I assumed with this setup).
Thanks for the information. Basically the main purpose of using coordinate node for ingestion is to distribute all the data coming for ingestion into all 50 data nodes, though it is just a theoretical idea. Is there any possible way to distribute all the data coming for ingestion uniformly to all 50 nodes, so that ingestion will get faster.
Also I want to understand the logic behind how data nodes get selected while creating index manually through PUT command, and is there any possible way to manipulate those node selection.
in order to do that you would need to know where the shards are and the idea is that you do not need to know. This would also imply that you would be able to react on shards being moved around. Again something the application should not care about.
Regarding the PUT command, I think your question is, how a shard gets chosen for a certain document. See https://www.elastic.co/guide/en/elasticsearch/reference/7.3/mapping-routing-field.html for an explanation and also how to influence this behaviour.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.