Marvel search latency does not match query time

[Maxwell_Flanders]

I have been testing the length of queries using the linux time command, ex.

time curl -s "IPADDRESS:9200/INDEX/_search?pretty" -d{ES_QUERY}

Some queries appear on the command line end to take anywhere from 1 to 10 seconds. However, when looking at the marvel stats, there has never been a search latency of over 1000ms, and they typically stay under 500ms, even when queries on my end appear to be taking a long time.

What might account for this discrepancy?? I'd at least expect some of the long queries to show up as spikes on the search latency chart.

Thank you!

[pickypg (Chris Earle)]

Hi @Maxwell_Flanders,

Query latency is calculated on a per shard basis. This means that, while the overall search may be slow, it may be relatively fast at the per shard level.

Can you show the header of the response that you're looking at? Particularly:

{
  "took": 12345,
  "_shards": { ... }

[Maxwell_Flanders]

@pickypg
Sure, here is the header for a query that took ~16s

  "took" : 16928,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },

Similarly, I have my es_slowlog settings set to report any query that takes over 1 ms. when I run these queries that appear to take a long time, I still never see the slowlogs reporting a time higher than 1000ms.