Elasticsearch same query has different performance

[ans76]

I have 2 nodes in my cluster.
n1: [master:true data:true] 8GB RAM 500GB Disk space
n2: [master:false data:true] 8GB RAM 500GB Disk space

Only one index for test(about 20000 docs), 5shards, 1replica
Run the same query 10000+ times in one thread, cost of time like:

1. 5ms
2. 13ms
3. 4ms
4. 12ms
5. 4ms
6. 13ms
   ....

Odd tests perform much better than even tests, any idea about this?

[eedugon (Eduardo González de la Herrán)]

Hi @ans76,

From where are you running the tests and towards what node? Each time towards one of the nodes? Have you tried running 100 times the same query towards the same node and then switching to another node?

Also take a look to the query and the data you are retrieving. Check the size of the documents and data, as maybe the JVM is doing garbage collections and you need to adapt the HEAP size every 2 requests, not sure.

You can check the amount of GCs via _node/stats API (_node/stats/jvm?pretty I guess).

Hope this is helpfull...

Regards and good luck!
Eduardo

[ans76]

Thanks for your reply. I run all the tests use one of nodes as coordinate node that get above results. and then switch to another node didn't change at all, and also gc is stable.

[ans76]

@eedugon There's one more thing, node n1 ping n2 cost 1.5ms on average, so does switch, any chance may network issue incurred?

[eedugon (Eduardo González de la Herrán)]

Hi @ans76,

About your network concern, I don't believe there's any problem with the 1.5ms reply on average, but it could be.
Anyway the topic is interesting, I would suggest to do the following tests:

• Do the searches with profile activated, to see where exactly the time is being consumed, and check the differences between the short and long ones.
  - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
• Try to force the search to use local data when possible instead of finding the data in the shards of other nodes (just for investigation purposes, to see if we get always the same response time when we ask the node to return all the data from itself. In a real traffic case I wouldn't recommend to use the preferred_local option).
  - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html
• Check with an index of 4 shards instead of 5.

Regards!
Eduardo