Elastic Logging - How to Trace a Request Across Multiple Remote Clusters (v8.10.2)

Elastic Stack Elasticsearch
1

We have a cross-cluster setup with the following components:

  • 2 coordinator clusters

  • Multiple remote (data) clusters

We’ve enabled slow logs on the remote clusters to capture details such as:

  • Search execution time

  • Number of hits

  • Query info per node

However, since the slow logs are generated per node, and a single search request can span multiple clusters, it becomes difficult to trace a request end-to-end.

Questions:

  1. What is the recommended way to trace a single request across clusters, given that logs are distributed across multiple nodes and clusters?

  2. Is there a way to aggregate or centralize logs, so we can correlate data for a single request more easily?

  3. Is there a better way to log search/index requests, especially in a multi-cluster setup?

Note:
We currently cannot enable audit logging, as security features are disabled in our environment.
Elastic version: 8.10.2

2

Hi @sakshijain

You’re running into a common challenge with cross-cluster setups: slow logs are per-node, so tracing a request end-to-end requires a consistent identifier. The easiest solution is to use the X-Opaque-Id HTTP header. Include a unique ID (e.g., a UUID) in every request — Elasticsearch propagates it across nodes and clusters, and it appears in slow logs. You can then search or filter logs by that ID to trace a single request.

For aggregating logs, the recommended approach is to centralize them using Filebeat or Elastic Agent. Ship slow logs from all clusters into a single monitoring cluster or index. This allows you to correlate entries across nodes and clusters easily and build Kibana dashboards for end-to-end visibility.

If you want deeper insights, you can supplement slow logs with the _tasks API for real-time task tracking. Later, enabling Audit Logging and Elastic APM can provide fully automated distributed tracing. But for now, the combination of X-Opaque-Id + centralized slow logs + _tasks API is the most practical way to trace and analyze requests in a multi-cluster setup without security enabled.