How to set Thread Pool settings for all data nodes in 7.14 Elastic cloud

Elastic Stack Elasticsearch
1

Hi,

We have 200K video assets on Elastic cloud, consuming 500mb of space on single node. Configuration wise we have
32 GB RAM Elasticsearch in 3 Zone
Dedicated Master with 1GB RAM | 8 vCPU | 3 Zone
Coordinating node with 1GB RAM | 4VCPU | 3 Zone
Kibana node 1GB RAM | 8 vCPU | 2 Zone
Enterprise Search nodes 8 GB RAM | Up to 8 vCPU | 2 Zone

We are targeting our application with load of 13 k request in 1 min for response time 5Oms per request.

However our request are getting rejected/failed from elastic data node and getting response time
min of 1ms
max 25 sec.

image
image1868×583 70.5 KB

My questions are:
Is this configuration is sufficient for load we are targeting?
Can we set the thread pool size of each node in Elastic cloud?

image
image1906×847 105 KB

2

Hi @rohitnagarro

Since you are using Elastic Cloud, you have support perhaps open a support ticket through the Elastic Cloud Console would be a good suggestion.

Also is that instance 000000032 the coordinating node? If so 1GB is pretty small perhaps scale it up to 2GB x 3 Zones . All searches and ingests will pass through those coordinating nodes.

3

You state that the data set is 500MB in size but the cluster hold over 66GB of data. What is the size of the index/indices you are querying? How many primary and replica shards does it have?

4

Hi @stephenb,
Thanks for your reply , I tried the support and however as per them "our current subscription level offers our world-class break/fix support, including general guidance and best practices. Although this question appears to be of a consultative nature outside of your subscription’s scope."

Instance 000000032 is the "data node", will try with 2 GBx 3 Zones.

5

Hi @Christian_Dahlqvist ,

I mean the index we are searching on is ~500Mb in size and growing. there are other indices(total 126, 13 created by us, rest are related to actastic and other elastic internal) also and no search is happing on them when I am testing.

Index under test is the one whose size is ~500MB rest indies created by us are of few KBs are has 2 primary and 1 replica.

6

How many primary shards does this index have?

7

2 primary shards

8

That is a small amount of data so I would recommend reindexing that into an index with a single primary shard. This should improve your handling of concurrent queries, especially if you have enough operating system page cache so the full index can be cached.

10

Any other solution I can try?

11

Thanks, I have tried with single primary shards, however response time we are getting is
~3 sec for 100 request in 4 sec.
and Min: 18sec Max: 26.5sec for 1000 request in 30 sec

Our target is 13k request in 1 min

I am using Elasticsearch Search API "https://[deploymentname].[region].gcp.cloud.es.io:9243/[index]/_search"

12

Your coordinating node is quite small so increasing its size to see if that makes any difference might be useful.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.