# Jaeger dropping spans with Elasticsearch backend in Kubernetes

**URL:** <https://discuss.elastic.co/t/jaeger-dropping-spans-with-elasticsearch-backend-in-kubernetes/294756>\
**Category:** APM\
**Tags:** docker, server\
**Created:** [January 19, 2022, 2:29am UTC](https://discuss.elastic.co/t/jaeger-dropping-spans-with-elasticsearch-backend-in-kubernetes/294756 "2022-01-19T02:29:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kashokan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kashokan/32/100501_2.png) [@kashokan](https://discuss.elastic.co/u/kashokan)\
**Post date:** [January 19, 2022, 2:29am UTC](https://discuss.elastic.co/t/jaeger-dropping-spans-with-elasticsearch-backend-in-kubernetes/294756/1 "2022-01-19T02:29:48Z")

</div>

Hello,

I have installed Jaeger with an Elasticsearch storage backend in Kubernetes (AWS EKS). I have 8 r5.4xlarge nodes and Elasticsearch version 7.16.2 ([docker.elastic.co/elasticsearch/elasticsearch:7.16.2](http://docker.elastic.co/elasticsearch/elasticsearch:7.16.2)). The deployment is done using the Elasticsearch helm chart ([helm-charts/elasticsearch at main · elastic/helm-charts · GitHub](https://github.com/elastic/helm-charts/tree/main/elasticsearch)). The values I have overridden are:

```auto
  elasticsearch:
    replicas: 4
    minimumMasterNodes: 3
    volumeClaimTemplate:
      accessModes: ['ReadWriteOnce']
      resources:
        requests:
          storage: 2000Gi
    resources:
      requests:
        cpu: '5000m'
        memory: '32Gi'
      limits:
        cpu: '8000m'
        memory: '32Gi'
    esJavaOpts: '-Xmx16g -Xms16g'

```

I have been seeing periodic dropping of spans, correlating with a spike in in-queue and save latency. Save latency, particularly is an indication that the storage backend isn't able to keep up. Additionally, the spikes correlate with an increase in flush operation timing. Also, the index write's memory steadily increases until the flush, when it drops and we have a drop in spans and spikes in latency. I don't have much experience with Elasticsearch, so any help/guidance is appreciated!

I have attached some screenshots of our Jaeger and Elasticsearch dashboards.

 ![Screen Shot 2022-01-18 at 6.44.56 PM](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b40eb5fd3a64076c9167a50bec6a090500c6a2cf.png)  
 ![Screen Shot 2022-01-18 at 7.21.34 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/8/88c8c9fd25a3b170510f5304c27f4ac09a45320a.jpeg)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2022, 7:30am UTC](https://discuss.elastic.co/t/jaeger-dropping-spans-with-elasticsearch-backend-in-kubernetes/294756/2 "2022-01-19T07:30:15Z")

</div>

7.1 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![kashokan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kashokan/32/100501_2.png) [@kashokan](https://discuss.elastic.co/u/kashokan)\
**Post date:** [January 21, 2022, 10:07pm UTC](https://discuss.elastic.co/t/jaeger-dropping-spans-with-elasticsearch-backend-in-kubernetes/294756/3 "2022-01-21T22:07:33Z")

</div>

The issue was not Elasticsearch, but rather the way we were routing traffic from our OpenTelemetry collector fleet to our Jaeger collector fleet. We were routing using Kubernetes' internal routing `jaeger-collector.jaeger.svc.cluster.local:14250`. We noticed the traces were not distributed evenly across the jaeger collectors, so we swapped in an Application load balancer with a target group using the gRPC protocol version. The root cause for us was the way gRPC works. After the swap, we are not dropping spans, translog ops and size is tiny and the index write memory is steady and flat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 18, 2022, 10:07pm UTC](https://discuss.elastic.co/t/jaeger-dropping-spans-with-elasticsearch-backend-in-kubernetes/294756/4 "2022-02-18T22:07:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.