# Signals export failing frequently as more than 20 services are exporting signals to elk at the same time

**URL:** <https://discuss.elastic.co/t/signals-export-failing-frequently-as-more-than-20-services-are-exporting-signals-to-elk-at-the-same-time/365610>\
**Category:** Kibana\
**Tags:** elastic-stack-monitoring\
**Created:** [August 27, 2024, 10:39am UTC](https://discuss.elastic.co/t/signals-export-failing-frequently-as-more-than-20-services-are-exporting-signals-to-elk-at-the-same-time/365610 "2024-08-27T10:39:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Harshul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshul/32/122633_2.png) [@Harshul](https://discuss.elastic.co/u/Harshul)\
**Post date:** [August 27, 2024, 10:39am UTC](https://discuss.elastic.co/t/signals-export-failing-frequently-as-more-than-20-services-are-exporting-signals-to-elk-at-the-same-time/365610/1 "2024-08-27T10:39:17Z")

</div>

I'm currently working on a test project that involves multiple services implemented in Python, Java, React, and Node.js. These services utilize OpenTelemetry for observability. The architecture is designed such that:

1. **Agent Collectors** : Several services (e.g., Service 1, Service 2, Service 3) are using a common OTLP collector, referred to as an agent collector.
2. **Central Collector** : All these agent collectors forward their telemetry data (traces, metrics, logs) to a central OTLP collector.
3. **ELK Export** : The central collector then exports this data to Elasticsearch (ELK) for storage and analysis.

#### The Problem:

We are intermittently experiencing the following errors during the metric export process:

plaintext

Copy code

```auto
error	exporterhelper/queue_sender.go:125	Exporting failed. No more retries left. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "max elapsed time expired rpc error: code = DeadlineExceeded desc = context deadline exceeded", "dropped_items": 822}

info	exporterhelper/retry_sender.go:129	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "1.030954787s"}

```

#### Key Details:

- **Service Implementation** : The issue occurs across multiple services using the OTLP protocol.
- **Common Pattern** : The error indicates that the `max elapsed time` for retrying has expired and the context deadline has been exceeded. This results in data being dropped.

Given that the central collector is responsible for exporting telemetry data to Elasticsearch, I suspect this issue might be related to the ELK stack, particularly in handling large volumes of incoming telemetry data or network latencies.

#### Request for Assistance:

- **Root Cause** : Could this issue be related to Elasticsearch's ability to handle telemetry data exported from the central collector? If so, what might be the underlying cause?
- **Potential Fixes** : Are there any specific configurations or optimizations on the Elasticsearch side that we should consider to avoid these errors?
- **Further Diagnosis** : Any recommendations on how to further diagnose or mitigate this issue would be greatly appreciated.

Thank you for your assistance!

---

<div class="post-metadata">

**Author:** ![Harshul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshul/32/122633_2.png) [@Harshul](https://discuss.elastic.co/u/Harshul)\
**Post date:** [October 16, 2024, 7:19am UTC](https://discuss.elastic.co/t/signals-export-failing-frequently-as-more-than-20-services-are-exporting-signals-to-elk-at-the-same-time/365610/2 "2024-10-16T07:19:36Z")

</div>

Still waiting for a response on this !