Negative Search Latency in Marvel

Elastic Stack Elasticsearch
1

I'm using the free edition of Marvel with ES 2.3.3. Today I've spotted negative search latency in graph:

Pasted image914×197 11.3 KB

Any comments?

2

Woah, you must have awesome hardware!

(Actually I have nothing helpful to add, sorry :p)

3

Hi Zaar Hai,

I'd like to tell you that we're predictively pre-searching for you, but this is a visualization bug that comes from how we calculate the the latency:

Search Latency is calculated by dividing the derivative of the total search time by the derivative of the total search count. Index Latency is the same thing. The derivative is necessary because the stats reported from Elasticsearch itself are just counters since the node started, so each time bucket (x coordinate) gets this extra step applied to avoid a useless display for this type of data.

In the above screenshot, my guess is that you were restarting the node a bunch of times, thus allowing those counters to be reset, which means one of the derivatives was lower (negative) combined with the rest of the data. Please let me know if that's not the case because it would be good to chase down how you ended up getting so many negatives.

Thanks

4

Nice!

I was not restarting nodes - I've joined node into the cluster after it was
down for two days and after ES relocated shards on it, I've played with
shard relocation speed by altering exclude._ip params.
During this later interval I've saw the negative latency.

Thanks,
Zaar