Terrible low performance search despite enough resources

We have two 3-nodes Elasticsearch cluster. Despite these clusters have approximately similar resources and the indices are quite similar, the speed of search in one of them is tenfold slower than another. We spent a lot of time for troubleshooting but couldn't find what is the problem. The odd thing is that the faster cluster is in production and is under more load than the slower one that is in stage zone! Since the faster cluster has more load we suspected to cache, but it doesn't seem to be the problem after doing some tests to check its effect.

I checked the storage type of two clusters to ensure the faster one has not ssd and have seen that's not the case. I've also checked GET _nodes/os and GET _nodes/jvm. All things except for names, ips and small differences in resources are equal, only the using_compressed_ordinary_object_pointers property in faster cluster is false but in the slower is true. I couldn't find good enough documents about this property and its effect on search performance and am curious to know :
if this property can cause the problem?
is it safe to change it in slower cluster whiteout any side effect?
if this is safe, how can I change it?

and if this property is not the reason, what another thing is potential enough to be the reason of such a terrible low performance search?

Did you checked your jvm.options for -xms/-xmx? Are they configured to be %50 of your total memory? What is your Ram/CPU/Disk details. How many shards do you have per node?
Can you also check ram/cpu usage of java service in each node?

You need to share the specs of your cluster, how many CPUs, Memory and Disk each node has, what is the Java HEAP configuration, what is the type of the Disk.

Also, share the number of indices and shards in both cluster as well as the versions.

The compressed ordinary object pointers will depend on the Java Heap Size, it is explained in this part of the documentation and it should be true.

1 Like

What is the output from the _cluster/stats?pretty&human API?