What is @timestamp field exactly?

[ebuildy (Thomas Decaux)]

Kibana version: 7.13.2

Elasticsearch version: 7.13.2

APM Server version: 7.13.2

APM Agent language and version: nodeJS / RUM

We run a very busy web site, with continuous traffic, but as you can see from the image below:

We receive data by batch,

it looks like @timestamp field is the time elasticsearch index the data?
Or this represent the time apm-server receive the event?

This give us totally wrong data-viz on APM dashboard:

Should I disable "transaction metrics" feature?

Our apm-server.yaml:

apm-server:
  # https://www.elastic.co/guide/en/apm/server/7.13/transaction-metrics.html
  aggregation:
    transactions:
      enabled: true
      interval: 1m
      max_groups: 5000
  sampling:
    keep_unsampled: true
  queue:
    mem.events: 10096
    flush.timeout: 5s
    flush.min_events: 50
  max_procs: 4
output:
  file:
    enabled: false
  elasticsearch:
    enabled: true
    hosts: ["XXXXXXX"]
    bulk_max_size: 750
    worker: 4
    max_retries: 0
    username: "${ELASTICSEARCH_USERNAME}"
    password: "${ELASTICSEARCH_PASSWORD}"
    timeout: 10
    ssl:
      enabled: true
      verification_mode: certificate
      certificate_authorities:
      - /usr/share/infra/datahub/certs/ca.crt
    backoff:
      init: 1s
      max: 30s
    pipelines:
    - pipeline: "apm_user_agent"
    - pipeline: "apm_ingest_timestamp"
    - pipeline: "apm_remove_span_metadata"

kibana.yaml

xpack.apm:
            enabled: true
            searchAggregatedTransactions: always

[sqren (Søren Louv Jansen)]

Hi there,

it looks like @timestamp field is the time elasticsearch index the data?
Or this represent the time apm-server receive the event?

@timestamp is recorded by the APM agent when the event happens. There is also event.ingested which is created by an ingest pipeline upon arrival in Elasticsearch.

This give us totally wrong data-viz on APM dashboard:

Yes, it looks odd when "zooming in" since metrics are indexed in batches every minute. If you choose a larger time range this should not be a problem though.

Should I disable "transaction metrics" feature?

If you don't want to use metrics you can disable this, yes. But I'm curious whether you have enabled this in Kibana also? By default Kibana will use transactions instead of metrics, unless this setting has been modified:

xpack.apm.searchAggregatedTransactions: 'never'

never is the default. Changing it to always or auto will make Kibana prefer metrics over transactions.

[sqren (Søren Louv Jansen)]

Sorry, one correction: For metric documents @timestamp will be set by the APM Server - not by the APM agent.

[sqren (Søren Louv Jansen)]

I created a bug report for this: [APM] Add minimum bucket size when using metric powered ui · Issue #103661 · elastic/kibana · GitHub. Aiming to get this fixed for 7.14 but I can't promise anything.

[ebuildy (Thomas Decaux)]

thanks you for the insights!