Elastic Logging - How to Trace a Request Across Multiple Remote Clusters (v8.10.2)

Elastic Stack Elasticsearch
1

We have a cross-cluster setup with the following components:

  • 2 coordinator clusters

  • Multiple remote (data) clusters

We’ve enabled slow logs on the remote clusters to capture details such as:

  • Search execution time

  • Number of hits

  • Query info per node

However, since the slow logs are generated per node, and a single search request can span multiple clusters, it becomes difficult to trace a request end-to-end.

Questions:

  1. What is the recommended way to trace a single request across clusters, given that logs are distributed across multiple nodes and clusters?

  2. Is there a way to aggregate or centralize logs, so we can correlate data for a single request more easily?

  3. Is there a better way to log search/index requests, especially in a multi-cluster setup?

Note:
We currently cannot enable audit logging, as security features are disabled in our environment.
Elastic version: 8.10.2

3

Is there any way to log such information in the coordinator (local) cluster or somewhere, from which remote cluster (node /shard) the data is retrieved