# Logstash stop working silently (workers die out)

**URL:** https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722
**Category:** Logstash
**Created:** [June 4, 2020, 9:45am UTC](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722 "2020-06-04T09:45:37Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Aldin\_Osmanagic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aldin_osmanagic/32/69692_2.png) [@Aldin\_Osmanagic](https://discuss.elastic.co/u/Aldin_Osmanagic)
#### Post date: [June 4, 2020, 9:45am UTC](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722/1 "2020-06-04T09:45:37Z")

</div>

Hello,

At the customer site, we are using ElasticStack to gather syslogs from the network devices.  
Logstash is receiving around 4000 syslogs/sec. We are using Logstash version 6.8.8.

The problem is that every 2-5 hours Logstasha workers start to die out until every worker dies and no more syslogs are received.

![Screenshot at 11-37-52](https://us1.discourse-cdn.com/elastic/original/3X/4/1/41722cc16a5e014eef20e9de6b2e38307ff77e4f.png)

Logstash is on a dedicated server, Elasticsearch Master + Kibana is one server, and we have 2  
Elasticsearch node servers.

Logstash x 1  
CPU: 4 x Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz  
RAM: 8G  
DISK: 20G`

![Screenshot at 11-39-27](https://us1.discourse-cdn.com/elastic/original/3X/3/7/371c049a45ee3ac5c15dd640ac2159f46215d48f.jpeg)

Kibana + Elasticsearch Master x 1  
CPU: 4 x Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz  
RAM: 8G  
DISK: 20G  
OS: CentOS

Elasticsearch Node x 2  
CPU: 4 x Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz  
RAM: 16G  
DISK: 20G za OS i 800G za data  
OS: CentOS

![Screenshot at 11-40-20](https://us1.discourse-cdn.com/elastic/original/3X/7/4/749776f01c61880dc1feb63638120b380af11442.png)

> # more /etc/logstash/logstash.yml
> 
> pipeline.batch.size: 250  
> pipeline.batch.delay: 50  
> pipeline.unsafe\_shutdown: true  
> pipeline.workers: 30  
> path.data: /var/lib/logstash  
> config.reload.automatic: true  
> config.reload.interval: 10s  
> path.logs: /var/log/logstash

> # more /etc/logstash/jvm.options
> 
> -Xms7096M -Xmx7096M  
> -XX:+UseParNewGC  
> -XX:+UseConcMarkSweepGC  
> -XX:CMSInitiatingOccupancyFraction=75  
> -XX:+UseCMSInitiatingOccupancyOnly  
> -Djava.awt.headless=true  
> -Dfile.encoding=UTF-8  
> -Djruby.compile.invokedynamic=true  
> -Djruby.jit.threshold=0  
> -XX:+HeapDumpOnOutOfMemoryError  
> -Djava.security.egd=file:/dev/urandom

Can you please help me troubleshoot this issue?

Thank you.  
Reagards

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 4, 2020, 1:22pm UTC](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722/2 "2020-06-04T13:22:37Z")

</div>

What is logged in the logstash logs when the worker threads die?

---

<div class="post-metadata">

### Author: ![Aldin\_Osmanagic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aldin_osmanagic/32/69692_2.png) [@Aldin\_Osmanagic](https://discuss.elastic.co/u/Aldin_Osmanagic)
#### Post date: [June 5, 2020, 7:34am UTC](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722/3 "2020-06-05T07:34:50Z")

</div>

There is nothing consistent in the log. Very often there is nothing at all, sometimes there is "DNS: timeout on resolving address", sometimes"Received an event that has a different character encoding than you configured." but I don't think that those are related because there is no consistency.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 3, 2020, 7:35am UTC](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722/4 "2020-07-03T07:35:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.