We have two 3-nodes Elasticsearch cluster. Despite these clusters have approximately similar resources and the indices are quite similar, the speed of search in one of them is tenfold slower than another. We spent a lot of time for troubleshooting but couldn't find what is the problem. The odd thing is that the faster cluster is in production and is under more load than the slower one that is in stage zone! Since the faster cluster has more load we suspected to cache, but it doesn't seem to be the problem after doing some tests to check its effect.
I checked the storage type of two clusters to ensure the faster one has not ssd and have seen that's not the case. I've also checked GET _nodes/os and GET _nodes/jvm. All things except for names, ips and small differences in resources are equal, only the using_compressed_ordinary_object_pointers property in faster cluster is false but in the slower is true. I couldn't find good enough documents about this property and its effect on search performance and am curious to know :
if this property can cause the problem?
is it safe to change it in slower cluster whiteout any side effect?
if this is safe, how can I change it?
and if this property is not the reason, what another thing is potential enough to be the reason of such a terrible low performance search?