Cluster With Mulitple Indexes With Different Performance Requirements


(swarmee.net) #1

Heya All,

Like everybody (I think) we have an elastic cluster with multiple indexes. Also like most people, some indexes need to have good consistent response times for searching, while others can be much slower.

If you have a big cluster with more than 3 nodes then I see how the hot warm architecture helps with this - basically you just allocate the slow indexes and searches (using the query preference parameter) to the nodes with slow disks, and you allocate the higher performance searches to the high performance nodes severing up your indexes with the high performance requirements.

But lets say you just have three data nodes and you have one small index that has to be as fast as possible for requests for specific resources like this https://<hostname>:<port>/<index>/_doc/<documentId>/_source (like < 10ms) - serving up these resources is the main purpose of the website, and three larger indexes that support complex searches and can be as slow as 10 seconds to complete a request.

What would be the best way to support this type of requirement. I have increased the index query cache size to 20%. And disabled index query caching on the other indexes (that don't have a high performance requirement). What are the other ways to allocate more resources and caching to one index in a cluster.

On a related note - can you confirm if "search" requests in this format https://<hostname>:<port>/<index>/_doc/<documentId>/_source fall into the filter context and therefore get cached in the Node Query Cache. I think they do - however I would love somebody to confirm please.


(Lee Hinman) #2

Hi Johnny,

These do not fall into the filter context, because in this case the document is retrieved directly from the IndexSearcher. I would, however, recommend that you investigate setting the realtime option to false to reduce the refresh load assuming that you are indexing into this index.

See: https://www.elastic.co/guide/en/elasticsearch/reference/6.4/docs-get.html#realtime


(swarmee.net) #3

Thanks Dakrone,

I will look at the realtime parameter.

As you have not respond to the first part of the question - I guess there are no index level settings currently to allocation more resources to a specific index on a node. That's fine just was keen to know if there was.

Thanks again


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.