# TCP : Error: closed stream

**URL:** <https://discuss.elastic.co/t/tcp-error-closed-stream/64431>\
**Category:** Logstash\
**Created:** [October 31, 2016, 11:07am UTC](https://discuss.elastic.co/t/tcp-error-closed-stream/64431 "2016-10-31T11:07:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Towser](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Towser](https://discuss.elastic.co/u/Towser)\
**Post date:** [October 31, 2016, 11:07am UTC](https://discuss.elastic.co/t/tcp-error-closed-stream/64431/1 "2016-10-31T11:07:00Z")

</div>

Hello everybody,

I have been looking for a solution at my problem for days but without any success.  
I tried upgrading to the newly 5.0 stack, setting up 1 VM for every level ( logstash, elastic, kibana), tried to change configuration as well but nothing seems to work.

Basically, I want to syslog drain log from Heroku ([https://devcenter.heroku.com/articles/log-drains](https://devcenter.heroku.com/articles/log-drains)), send it to my logstash, process it and then send it to my elastic. Everything is working fine for a while until I got this error (which happens once a day) and I need to restart manually in order for logstash to be back running. When this error happens, my CPU and memory go very high (almost 100%) but are low on a normal use.

Here is my error :

```
 [2016-10-31T10:51:59,693][ERROR][logstash.instrument.periodicpoller.jvm] PeriodicPoller: exception {:poller=>#<LogStash::Instrument::PeriodicPoller::JVM:0x263e02b5 @task=#<Concurrent::TimerTask:0x35d23145 @observers=#<Concurrent::Collection::CopyOnNotifyObserverSet:0x46c55984 @observers={#<LogStash::Instrument::PeriodicPoller::JVM:0x263e02b5 ...>=>:update}>, @timeout_interval=60.0, @running=#<Concurrent::AtomicBoolean:0x4da42c1b>, @StoppedEvent=#<Concurrent::Event:0x69d874f3 @set=false, @iteration=0>, @execution_interval=1.0, @do_nothing_on_deref=true, @run_now=nil, @freeze_on_deref=nil, @executor=#<Concurrent::SafeTaskExecutor:0x53d1935 @task=#<Proc:0x759820b5@/usr/share/logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:52>, @exception_class=StandardError>, @StopEvent=#<Concurrent::Event:0x19b00ee9 @set=false, @iteration=0>, @value=nil, @copy_on_deref=nil, @dup_on_deref=nil>, @peak_threads=1030, @peak_open_fds=4095, @metric=#<LogStash::Instrument::Metric:0x33e59ec0 @collector=#<LogStash::Instrument::Collector:0x10e9bbac @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x5efcdd38 @store=#<Concurrent::Map:0x6a55d3da @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x730b4f3f>, @fast_lookup=#<Concurrent::Map:0x6c07cc00 @default_proc=nil>>, @observer_state=false, @snapshot_task=#<Concurrent::TimerTask:0x542ac9a5 @observers=#<Concurrent::Collection::CopyOnNotifyObserverSet:0x10a50bf9 @observers={#<LogStash::Instrument::Collector:0x10e9bbac ...>=>:update}>, @timeout_interval=600.0, @running=#<Concurrent::AtomicBoolean:0x6bcb6096>, @StoppedEvent=#<Concurrent::Event:0x69aeebab @set=false, @iteration=0>, @execution_interval=1.0, @do_nothing_on_deref=true, @run_now=nil, @freeze_on_deref=nil, @executor=#<Concurrent::SafeTaskExecutor:0x516cdfa4 @task=#<Proc:0x1e3eac08@/usr/share/logstash/logstash-core/lib/logstash/instrument/collector.rb:87>, @exception_class=StandardError>, @StopEvent=#<Concurrent::Event:0x2ec1a089 @set=false, @iteration=0>, @value=false, @copy_on_deref=nil, @dup_on_deref=nil>>>, @options={:polling_interval=>1, :polling_timeout=>60}>, :result=>nil, :exception=>#<Concurrent::TimeoutError: Concurrent::TimeoutError>, :executed_at=>2016-10-31 10:51:59 +0000}
[2016-10-31T10:51:59,721][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Tcp type=>"heroku", port=>1514, codec=><LogStash::Codecs::Plain id=>"plain_7d17f778-6aa6-47f4-93f4-e97f904aca21", enable_metric=>true, charset=>"UTF-8">, ssl_enable=>true, ssl_verify=>false, ssl_cert=>"/etc/pki/tls/certs/logstash-forwarder.crt", ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key", id=>"3d22eb2595295c0b2bedecb3456ee342ca36f418-1", enable_metric=>true, host=>"0.0.0.0", data_timeout=>-1, mode=>"server", ssl_key_passphrase=><password>>
  Error: closed stream

```

My setup is

- Ubuntu 16.04
- openjdk version "1.8.0\_91"
- ELK version 5.0

Logstash input config file (I tried with codec line as well):

```
input {
    tcp {
        type => "heroku"
        port => 1514
	codec => "plain"
        ssl_enable => true
        ssl_verify => false
        ssl_cert => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    }
}

```

Did someone had a similar issue before ?

Thanks for your help

---

<div class="post-metadata">

**Author:** ![Towser](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Towser](https://discuss.elastic.co/u/Towser)\
**Post date:** [November 3, 2016, 2:11am UTC](https://discuss.elastic.co/t/tcp-error-closed-stream/64431/2 "2016-11-03T02:11:32Z")

</div>

Anyone ? It is really blocking me for Production system and I can't get it work. I even tried to remove all my filters.  
My VM are on Azure.

---

<div class="post-metadata">

**Author:** ![Towser](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Towser](https://discuss.elastic.co/u/Towser)\
**Post date:** [November 14, 2016, 2:36am UTC](https://discuss.elastic.co/t/tcp-error-closed-stream/64431/3 "2016-11-14T02:36:45Z")

</div>

I tried disabling ssl, it looks like it runs longer but still hang after a while.

Here is the log output a bit more verbose :

> [2016-11-14T02:34:34,372][DEBUG][logstash.pipeline] Pushing flush onto pipeline  
> [2016-11-14T02:34:34,476][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::Tcp type=\>"heroku", port=\>1514, codec=\>\<LogStash::Codecs::Multiline pattern=\>"^10094", what=\>"next", id=\>"de27c6ee416913e9951f48707d2512bfea049672-1", enable\_metric=\>true, negate=\>false, charset=\>"UTF-8", multiline\_tag=\>"multiline", max\_lines=\>500, max\_bytes=\>10485760\>, id=\>"de27c6ee416913e9951f48707d2512bfea049672-2", enable\_metric=\>true, host=\>"0.0.0.0", data\_timeout=\>-1, mode=\>"server", ssl\_enable=\>false, ssl\_verify=\>true, ssl\_key\_passphrase=\>\>  
> Error: closed stream  
> Exception: IOError  
> Stack: org/jruby/ext/socket/RubyTCPServer.java:148:in `accept' /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.0.3/lib/logstash/inputs/tcp.rb:116:in `run\_server'  
> /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.0.3/lib/logstash/inputs/tcp.rb:88:in `run' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `inputworker'  
> /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:325:in `start\_input'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/tcp-error-closed-stream/64431/4 "2017-07-06T04:30:35Z")

</div>