Two same nodes but different stats

Elastic Stack Elasticsearch
1

Hello, I have a cluster with replica =1. So, I have 2 nodes which serve same shards. BigDesk shows that both of them have similar number "Search requests per second", but "Search time per second" number on the second node is about twice larger. Also CPU consumption is higher on the second node. What could it be? Where could I start troubleshoot the problem?

Thanks,
Eugene

2

How are you indexing and searching into the cluster, are you using load balancing?
Take a look at hot_threads and GC as well.

3

For indexing and searching I'm using Java API client it connects via transport protocol (port 9300).
GC shows about the same numbers as well.
I don't use load balancer, the client has all the nodes seeded and sniff property is true.
Similar number of requests per second makes me think that the client distributes requests evenly. But "Search time per second" probably says that the second node slower for some reason, even though they are the same hardware, and built from the same image.
I ran hot_threads many times, I don't understand how it could help, I see that search is one of the busiest thread, which I guess is the expected behavior. Here is an example:

Thanks,
Eugene