# Logstash is not load-balanced properly

**URL:** <https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021>\
**Category:** Logstash\
**Created:** [April 27, 2020, 8:01pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021 "2020-04-27T20:01:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\
**Post date:** [April 27, 2020, 8:01pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021/1 "2020-04-27T20:01:35Z")

</div>

Hi,

We are using logstash version: 7.6.1, from the helm chart - [https://github.com/elastic/helm-charts/tree/master/logstash](https://github.com/elastic/helm-charts/tree/master/logstash).

resources:

```
  requests:
    cpu: "2"
    memory: "6Gi"
  limits:
    cpu: "2"
    memory: "6Gi"
logstashJavaOpts: "-Xmx3g -Xms3g"
logstashPersistenceStorage: 6Gi
logstashReplicaCount: 2

```

logstash config:

```
  logstash.yml: |
      http.host: 0.0.0.0
      config.reload.automatic: "true"
      queue.type: persisted
      queue.checkpoint.acks: 0
      queue.checkpoint.writes: 0
      queue.checkpoint.interval: 0
      queue.drain: "true"
      queue.max_bytes: 3gb # disk capacity must be greater than the value of `queue.max_bytes`
      pipeline.workers: 4
      pipeline.batch.size: 5000
      #pipeline.batch.delay: 50
      # X-Pack
      http.host: "0.0.0.0"
      xpack.monitoring.enabled: true

```

kubernetes logstash service is behind an AWS internal ELB, for some reason all the requests to logstash go to only on logstash pod and the second pod is mostly idle.  
here's the ELB configuration:

```
service:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: logstash.{{ .Environment.Values.dnsZoneName }}
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: {{ .Environment.Values.logstashSecurityGroup }}
  type: LoadBalancer
  ports:
    - name: beats
      port: 5044
      protocol: TCP
      targetPort: 5044
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080

```

`This happens in all environments`  
here is a screenshot of CPU:

 ![Screen Shot 2020-04-27 at 12.49.25 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de13a8eadae0d1d5e6b657357c0966a157fe6b91.png)  
As you can see the blue line is for `logstash-0` and purple is for `logstash-1` pod, at any given point of time only `ONE POD IS USED`. This doesn't happen with any other services, we use the same networking for kibana but there are no load-balancing issues there.

`Events rate screenshots:`

logstash-0:

 ![Screen Shot 2020-04-27 at 12.59.05 PM](https://us1.discourse-cdn.com/elastic/original/3X/2/3/23c0fc05f8e52be29464edc213d2a37d04ebd2ee.png)

logstash-1:

 ![Screen Shot 2020-04-27 at 12.59.12 PM](https://us1.discourse-cdn.com/elastic/original/3X/6/7/6718a2bfa88998f8410aa99fe55ab694ddaf87b8.png)

`One more question:`  
we give 6GB of RAM for logstash pods, how much of that can be allotted for HEAP space, right now we set it to 3GB of heap space, is that appropriate or do we need to adjust it?  
Based on what values should we adjust the heap space?

here's a heap space graph for the pod which is used most of the time:

 ![Screen Shot 2020-04-27 at 12.55.36 PM](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b21c15cd63f9ab4b740de1766f660f2419a0e6be.png)

---

<div class="post-metadata">

**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\
**Post date:** [April 29, 2020, 6:34pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021/2 "2020-04-29T18:34:01Z")

</div>

any update here please?

- Thank you

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [April 29, 2020, 6:48pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021/3 "2020-04-29T18:48:47Z")

</div>

Let's address the first problem.

If I correctly understand, 2 Logstash instances receive events from Filebeat or Metricbeat passing through an Amazon load balancer.

Can you please share the `filebeat.yml` file?  
In particular, the Logstash output section.

In the [docs](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html) we suggest using a `ttl` to force Filebeat to reconnect as Filebeat uses TCP persistent connections.

In particular, you can try to add:

```auto
pipelining: 0
ttl: 1m

```

---

<div class="post-metadata">

**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\
**Post date:** [April 29, 2020, 7:33pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021/4 "2020-04-29T19:33:33Z")

</div>

disabling pipelining and enabling TTL solved our problem.

Let's move on to the second question please:

```auto
we give 6GB of RAM for logstash pods, how much of that can be allotted for HEAP space, right now we set it to 3GB of heap space, is that appropriate or do we need to adjust it?
Based on what values should we adjust the heap space?

```

- Thank you

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [April 29, 2020, 7:49pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021/5 "2020-04-29T19:49:21Z")

</div>

The JVM Heap size has to be sized depending on:

- the number of parallel pipelines
- the batch size for each pipeline
- the number of workers per pipeline
- the average of the events being processed
- the filters you're using (e.g. some filters cache values on memory)
- the number of connections open (both input & output connections)
- many other variables

To conclude, I would enable the Stack Monitoring as you've done and I would check the frequency of JVM Garbage Collections & the GC durations.  
Also ensure you're using the `jvm.options` file shipped with Logstash (excluding the size of the JVM Heap).

Other suggestions:

- Profiling the JVM Heap [https://www.elastic.co/guide/en/logstash/7.6/tuning-logstash.html#profiling-the-heap](https://www.elastic.co/guide/en/logstash/7.6/tuning-logstash.html#profiling-the-heap)
- Heap sizing [https://www.elastic.co/guide/en/logstash/7.6/heap-size.html](https://www.elastic.co/guide/en/logstash/7.6/heap-size.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 27, 2020, 7:49pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021/6 "2020-05-27T19:49:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.