How to use traces with the intake API

[LordMathis (Matt Irwin)]

Kibana version: 7.3

Elasticsearch version: 7.3

APM Server version: 7.3

APM Agent language and version: New agent built for all versions of Delphi

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Web download

I'd like to preface this discussion with the knowledge that I don't exactly quite understand what traces are used for. I believe it is some kind of method to clump all your transactions together so you can observe the flow of the entire application not just the flow of your transaction.

My issue is essentially this, the intake API docs don't specify how to properly create a Trace, so I assumed it would be as simple as generating a trace ID and passing that trace id in on all my JSON, but it would appear as though this does not work. When I go to my traces tab there is nothing there and the % of trace on the transaction is N/A.

What is it that I need to do to properly create a trace using the intake API?

[Sergey_Kleyman]

I don't exactly quite understand what traces are used for

Traces are very useful for applications composed of more than one layer/service. You can have an agent on each layer/service and each agent will produce a transaction but trace ties these transactions together by using distributed tracing. One of the mechanisms for distributed tracing is to pass tracing context from agent of the caller service to agent of the callee service. For example by using dedicated HTTP headers. Please note that official Elastic APM Agents use different header names but it might change soon.

the intake API docs don't specify how to properly create a Trace

As you correctly noted Elastic APM Intake API doesn't have an event for trace. This is because trace is an ephemeral entity. It's not like other events (transaction, span, etc.) that are produced by one of the agents and sent to APM Server. Trace is a composed of transactions that have the same trace_id property and the way agents decide on which trace_id to assign to each transaction is based on distributed tracing I mentioned above. If the transaction is the first one (i.e., there was no incoming tracing context) then the agent will generate a new trace_id, otherwise agent will use trace_id from the incoming tracing context and in any case the agent will propagate trace_id to downstream (AKA "callee") agents. In addition transaction event has parent_id which also arrives via incoming tracing context and determines where the transaction should be placed in the trace's tree of transactions.