# Logstash stopped logging, but I don't see errors

**URL:** <https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538>\
**Category:** Logstash\
**Created:** [February 17, 2017, 5:31pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538 "2017-02-17T17:31:27Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\
**Post date:** [February 17, 2017, 5:31pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/1 "2017-02-17T17:31:27Z")

</div>

I have an ELK stack. Logstash v.2.3.4 and Kibana v4.5.4 services are running on one server. I have another box running Elasticsearch v2.4.1 in client mode, and then a 3-box Elasticsearch v2.4.1 cluster running in master and data mode. All was working until yesterday, that is I was able to see the logs I've been sending to the stack.

For some reason, I don't see any logs today (I filter out "Today" in Kibana). I don't see my logstash.stdout file increasing size (I use the stdout plugin in my logstash.conf output filter). The only logstash log I see are these. (I have snipped the log since I can't post all of it in this message). Via tcpdump, I can tell that I'm receiving logs, and sending them out to my Elasticsearch client. What am I missing?

I have restarted my logstash service, and the elasticsearch service on the client box.

{:timestamp=\>"2017-02-17T17:10:01.494000+0000", :message=\>#\<LogStash::PipelineReporter::Snapshot:0x567d9997 @data={:events\_filtered=\>1292377, :events\_consumed=\>1292377, :worker\_count=\>4, :inflight\_count=\>13, :worker\_states=\>[{:status=\>"run", :alive=\>true, :index=\>0, :inflight\_count=\>2}, {:status=\>"sleep", :alive=\>true, :index=\>1, :inflight\_count=\>4}, {:status=\>"sleep", :alive=\>true, :index=\>2, :inflight\_count=\>1}, {:status=\>"sleep", :alive=\>true, :index=\>3, :inflight\_count=\>6}], :output\_info=\>[{:type=\>"stdout", :config=\>{"codec"=\>"rubydebug", " **ALLOW\_ENV**"=\>false}, :is\_multi\_worker=\>false, :events\_received=\>1292377, :workers=\>\<Java::JavaUtilConcurrent::CopyOnWriteArrayList:485794817 [\<LogStash::Outputs::Stdout codec=\>\<LogStash::Codecs::RubyDebug metadata=\>false\>, workers=\>1\>]\>, :busy\_workers=\>0}, {:type=\>"elasticsearch", :config=\>{"user"=\>"logstash", "password"=\>"l0gst@sh", "ssl"=\>"true", "ssl\_certificate\_verification"=\>"true", "truststore"=\>"/etc/elasticsearch/truststore.jks", "truststore\_password"=\>"pw", "hosts"=\>["[https://10.53.162.51:9200](https://10.53.162.51:9200)"], "index"=\>"logstash-eu-%{customer}-%{+YYYY.MM.dd}", " **ALLOW\_ENV**"=\>false}, :is\_multi\_worker=\>false, :events\_received=\>1292377, :workers=\>\<Java::JavaUtilConcurrent::CopyOnWriteArrayList:-1478703875 [\<LogStash::Outputs::ElasticSearch user=\>"logstash", password=\>, ssl=\>true, ssl\_certificate\_verification=\>true, truststore=\>"/etc/elasticsearch/truststore.jks", truststore\_password=\>, hosts=\>["[https://10.53.162.51:9200](https://10.53.162.51:9200)"], index=\>"logstash-eu-%{customer}-%{+YYYY.MM.dd}", codec=\>\<LogStash::Codecs::Plain charset=\>"UTF-8"\>, workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, flush\_size=\>500, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, retry\_on\_conflict=\>1, (snip)  
`pop'"}]}\>, :level=\>:warn}

---

<div class="post-metadata">

**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\
**Post date:** [February 17, 2017, 5:48pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/2 "2017-02-17T17:48:51Z")

</div>

All boxes are 4-core/16GB mem machines BTW, and I setup Elasticsearch to use 50% of the memory where the service is running.

---

<div class="post-metadata">

**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\
**Post date:** [February 17, 2017, 7:52pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/3 "2017-02-17T19:52:08Z")

</div>

I restarted the server, and I don't accumulate the logs anymore. However, Logstash just stops capturing logs. I have the following input and output filter in my configuration file. I see that /var/log/logstash/logstash.stdout stops growing after a few seconds. To remove Elasticsearch out of the equation, i removed it from the output filter.

```
input {
  udp {
    port => 5514
    codec => json
  }
}

filter {
  grok {
    ...
  }
}

output {
  stdout {
    codec => rubydebug
  }
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 18, 2017, 10:12pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/4 "2017-02-18T22:12:53Z")

</div>

Is there still data coming in via udp?

Can you upgrade?

---

<div class="post-metadata">

**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\
**Post date:** [February 20, 2017, 3:07pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/5 "2017-02-20T15:07:57Z")

</div>

Yes I still see data coming in, per tcpdump. This was working fine, but stopped working Thursday of last week. Upgrading is my last resort since I want to understand the problem.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 20, 2017, 8:22pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/6 "2017-02-20T20:22:46Z")

</div>

Why is upgrading a last resort, you're running a relatively old version.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 20, 2017, 8:23pm UTC](https://discuss.elastic.co/t/logstash-stopped-logging-but-i-dont-see-errors/75538/7 "2017-03-20T20:23:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.