My performance won't scale with the cluster

[martward (Martijn)]

Hi all,

I'm trying to get my cluster to scale so I can handle more requests. I used to have 2 ES nodes on 1 K8s nodes and 2 indexes with 2 shards and 1 replicas.
I tried increasing the number of ES nodes to 4 and increasing the number of K8s nodes to 2, and the number of replicas to 3. I've been playing around with the shards and replicas, but I don't notice any performance increase whatsoever. The cluster is set up using ECK.
I see no difference if I'm using 1 shard and 3 replicas or 1 shard and 1 replica.

I have an async Elasticsearch client running in a python application which uses the elastic-es-http service as the host. Since I'm connected to the http service I can't (and I assume don't need to) sniff the nodes on startup.

I can handle about 14 requests per second, every request I do fires 7 queries at Elasticsearch simultaneously using the msearch function.

Am I missing something obvious that prevents it from scaling?

Thank you

[leandrojmp (Leandro Pereira)]

You need to provide more information.

Which performance you want to improve? Indexing performance? Querying performance?

What are the specs of your nodes and what is the size of your index?

[martward (Martijn)]

Hi, thanks for your time.

The performance I'm trying to improve is that of the querying. I noticed the indexing does get faster if I have more shards, which makes sense. The querying seems not to scale at all so far though.
The two kubernetes nodes have 4 CPUs and 16gb each. I have 4 elastic nodes running on there, each node having a limit of 1.5 cpu and 6gb memory. Under full load I see a node using 2.5gb of memory and abou 0.4 CPU. The heap size is set to 2gb.
If I check the usage, I don't see any hardware limitations.
I have multiple indices (indexes?), two of them are used for search, one for logging and one for keeping track of something. The latter two are not that important at the moment, both have one shard and one replica. The other two I am trying to make faster. One is 1.3gb and the other about 400mb. I have been playing with he shards and replicas of those, but so far I don't see any changes in performance, for better or for worse.

Again, thanks for taking the time

[martward (Martijn)]

One thing I think that could be the (or part of) the issue is how I connect to the cluster. There are 4 nodes which have the same configuration, they are simply called 'default'. So I have an elast-es-default-[0,1,2,3]. There is one service called elastic-es-http which is what I connect the python client to. I know you can provide multiple hosts, but I only get one service so I can't specify the individual hosts for the client. I figured that the load balancing would be done by this clientIP, but I'm starting to wonder whether that is incorrect.