Is there a way to make the data ingest load separate from the query aggregation activity? My goal is to allocate some data nodes that can handle the ingest and low level query load. But for higher level aggregation for queries, how do I specify that that activity should be directed to nodes that I have reserved for this purpose?
I don't think you can as an index that needs to be updated will be updated on every shard that is holding the data.
The only solution I can see which is far from being near real time is to index in one cluster and do snapshot/restore every x minutes to copy the data to the "query only cluster"...
But could you tell what exact problem you are trying to solve? I mean "do you have a problem right now?"
I have some nodes that need to split since memory allocation is way over 32GB, like 3X. CPU goes nuts with GC under any kind of load. I am thinking that I will create some 'coordinating' node processes to get back some of the memory for User Query, or large Aggregations specifically, and shrink all nodes below the 32GB threshold.
When data for a query is fetched from many shards on different nodes to a single point for further aggregation, does this happen in the coordinating node ?
My indexes are write once, ready many and with heavy and constant inflow.
You should not allocate more than 30gb of HEAP to a node.
You can definitely start more than one node if this is needed on the same machine. Just make sure that primaries and replicas are not allocated within the same physical machine. Read Shard allocation awareness | Elasticsearch Guide [8.11] | Elastic
Having coordinating node and master nodes would help IMO. But then just use the coordinating nodes for indexing and querying. You can launch some coordinating nodes for index operations and some others for search operations.
When data for a query is fetched from many shards on different nodes to a single point for further aggregation, does this happen in the coordinating node ?
Aggregation is happening on each shard but then the result is merged in the coordinating node.
Yes, this is the problem that I am solving.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.