APM latency statistics - what is it exactly? why does it not align with "transaction.duration.us"?

[cozog]

Kibana version: 8.5.3
Elasticsearch version: 8.5.3
APM Server version: 8.5.3
APM Agent language and version: Java, 1.26.0

Hi,

We have APM visualizations that are showing us average latency:

now i am also trying to pull the average request time programmatically from the Elastic Search API like so:

endpoint: https://{host}/traces-apm-*/_search
request payload:

{
    "size": 0,
    "query": {
        "bool": {
            "filter": [
                {
                    "match": {
                        "service.name": "mdsgeoserver"
                    }
                },
                {
                    "match": {
                        "transaction.name": "Dispatcher"
                    }
                },
                {
                    "range": {
                        "@timestamp": {
                            "gte": "2023-12-08T00:00:00",
                            "lt": "2023-12-09T00:00:00"
                        }
                    }
                }
            ]
        }
    },
    "aggs": {
        "avg_duration": {
            "avg": {
                "field": "transaction.duration.us"
            }
        }
    },
    "track_total_hits": true
}

however the result differs than what i am seeing in the visualization tool;
response body:

{
    ...
    "aggregations": {
        "avg_duration": {
            "value": 982760.6078955772
        }
    }
}

so the API says the "average duration" is ~0.98 seconds, whereas the "latency" seen in the Kibana GUI is 1.295 seconds;

do these words mean different things? does the term "latency" mean something here other than when the transaction started, and when it ended?

or is my query suspect?

apologies if this is obvious or something i should know; I've been Googling and trawling through the documentation but nowhere is "latency" defined with clarity.

[stephenb (Stephen Brown)]

Hi @cozog Welcome to the community!

Great question... and you are on the right track ...

I think you have a simple issue...

@cozog

                    "range": {
                        "@timestamp": {
                            "gte": "2023-12-08T00:00:00",
                            "lt": "2023-12-09T00:00:00"
                        }
                    }
  

You range query does not have the time zone so you are probably not comparing the same timeframes....

It should be something like this, AND you need to convert the times to UTC...
My example I am in PST, so 8 hours behind... so these would be proper

{
          "range": {
            "@timestamp": {
              "gte": "2023-12-15T03:00:00.000Z", <!-- UTC
              "lt": "2023-12-15T04:00:00.000Z"
            }
          }
  
or

        {
          "range": {
            "@timestamp": {
              "gte": "2023-12-14T19:00:00.000-08:00", <!-- PST
              "lt": "2023-12-14T20:00:00.000-08:00"
            }
          }
        }

dates are always stored internally in Elasticsearch as ms (or sometimes nano secs) since the Epoch in UTC. And then they are displayed in Human Readable Format in Your Timezone in Kibana

If you were to look at the date format for the @timestamp the would be [strict_date_optional_time||epoch_millis]

Read out this here

When you run DSL it will use UTC unless you provide a Timezone...

[cozog]

yup that was exactly it, awesome i'd kiss you if i could