KQL Search for fields in APM traces metadata

[SUMANT_MISHRA]

Hi,

I am sending a custom header CorrelationId from UI while sending API requests. In APM, I can see the property in metadata list but if I try to search that field using KQL, it's not working.

I am trying to link all the requests starting from UI, application logs, traces (transactions) and response. If I go to particular transaction in APM then click on Metadata tab, I can see the custom header. But I am not able to find that transaction using KQL query.

As I found from google or other articles, we can add that field in "index.query.default_field" in Edit Settings section. But I can see there are many indexes like apm-7.10.1-transaction-000001 and there are total 7 indexes. I believe this is increasing based on the data.

So, my question is how we can make that field searchable. Below are the JSON structure for log and screenshot for metadata in APM:

Application Log JSON:

{
  "_index": "logs-development",
  "_type": "_doc",
  "_id": "PcJ_gHoBl0UAJkXaamBz",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-07-07T15:54:29.7688060+05:30",
    "level": "Information",
    "messageTemplate": "Token creation is started",
    "message": "Token creation is started",
    "fields": {
      "ClientIp": "127.0.0.1",
      "ClientAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
      "CorrelationId": "1625653467441"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-07-07T10:24:29.768Z"
    ]
  },
  "highlight": {
    "fields.CorrelationId": [
      "@kibana-highlighted-field@1625653467441@/kibana-highlighted-field@"
    ],
    "message": [
      "@kibana-highlighted-field@Token@/kibana-highlighted-field@ creation is started"
    ]
  },
  "sort": [
    1625653469768
  ]
}

Screenshot from APM Metadata:

Thanks

[forloop (Russ Cam)]

Hi @SUMANT_MISHRA,

APM transactions are persisted in a JSON structure that conforms to the Elastic Common Schema. The http request headers are stored in http.request.headers on the _source document, which you'd be able to target with a query with something like

GET apm-*-transaction/_search
{
  "query": {
    "term": {
      "http.request.headers.CorrelationId": {
        "value": "1625653467441"
      }
    }
  }
}

It looks like the CorrelationId might be serving a similar purpose to the W3C Trace Context, which the agent supports for distributed tracing. Log Correlation is supported with Serilog and NLog, and can be configured with other logging providers.