# Kubernetes Logstash statefulset under a Service of type Load Balancer (Amazon EKS) uneven distribution of events between pods

**URL:** <https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578>\
**Category:** Logstash\
**Created:** [January 5, 2023, 9:22pm UTC](https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578 "2023-01-05T21:22:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vikasp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikasp/32/90872_2.png) [@vikasp](https://discuss.elastic.co/u/vikasp)\
**Post date:** [January 5, 2023, 9:22pm UTC](https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578/1 "2023-01-05T21:22:41Z")

</div>

I am working with a self managed elasticsearch cluster hosted in Amazon EKS.  
The pipeline flow is:

1. filebeat agent is deployed in all ec2 servers seding data to logstash. [https://logstash.company.com:5046](https://logstash.company.com:5046).
2. Logstash is deployed as a statefulset. A service of type load balancer pointing to the pods of this statefulset. Route53 endpoint [logstash.company.com](http://logstash.company.com) -\> load balancer dns.
3. Logstash config is pipeline managed in kibana/es.
4. I see that there is a huge difference in the events received count for each pod, some of them are like 500m and some are like 50million for the same time from when they started.

Below are sample screenshots:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/a/2ab52c920f1268ee501f91cb0f794e918909c9cf.png)

service definition:

```auto
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
    service.beta.kubernetes.io/aws-load-balancer-type: elb
  labels:
    app: lg-endpoint-eks-filebeat-1
  name: lg-endpoint-eks-filebeat
  namespace: elasticsearch
spec:
  ports:
  - name: http-d
    nodePort: 31415
    port: 5046
    protocol: TCP
    targetPort: 5046
  - name: metrics
    nodePort: 30315
    port: 80
    protocol: TCP
    targetPort: 9600
  selector:
    app: lg-endpoint-eks-filebeat-1
  type: LoadBalancer

```

Not able to figure out why the way it is.  
Also, this service definition creates a Amazon Classic load balancer.

Does loadbalancer has some stickiness ? or does filebeat has anything to do with this behaviour ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 5, 2023, 9:32pm UTC](https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578/2 "2023-01-05T21:32:11Z")

</div>

> [@vikasp](#):
>
> Does loadbalancer has some stickiness ?

The loadbalancer balances connection attempts. If you have multiple clients they will often get distributed across multiple servers. If you have a single filebeat it will establish a connection to one of the servers and keep re-using it, so basically all the events go to one server until the client restarts.

---

<div class="post-metadata">

**Author:** ![vikasp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikasp/32/90872_2.png) [@vikasp](https://discuss.elastic.co/u/vikasp)\
**Post date:** [January 5, 2023, 9:37pm UTC](https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578/3 "2023-01-05T21:37:47Z")

</div>

I have 100s of agents connecting to logstash, Is there any way that I can distribute the load ?  
Does adopting ingress help instead of load balancer ?  
or any other approach ?

---

<div class="post-metadata">

**Author:** ![vikasp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikasp/32/90872_2.png) [@vikasp](https://discuss.elastic.co/u/vikasp)\
**Post date:** [January 8, 2023, 3:08pm UTC](https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578/4 "2023-01-08T15:08:20Z")

</div>

I found that we should be using the ttl config in filebeat.

> **[Configure the Logstash output | Filebeat Reference \[8.5\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_ttl)**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2023, 3:08pm UTC](https://discuss.elastic.co/t/kubernetes-logstash-statefulset-under-a-service-of-type-load-balancer-amazon-eks-uneven-distribution-of-events-between-pods/322578/5 "2023-02-05T15:08:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.