# Filebeat : Badly Stuck with Connection reset by peer error **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 22, 2016, 3:16pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878 "2016-11-22T15:16:48Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 3:16pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/1 "2016-11-22T15:16:49Z")
Hi, I'm badly stuck with connection reset by peer error even though all tcp ports are opened. Can anyone guide me here. **2016-11-22T13:32:39Z INFO Error publishing events (retrying): read tcp 10.3.2.20:35816-\>10.3.1.13:5044: read: connection reset by peer** **2016-11-22T13:32:39Z INFO send fail** **2016-11-22T13:32:39Z INFO backoff retry: 2s** ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 22, 2016, 3:39pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/2 "2016-11-22T15:39:26Z")
which versions of filebeat and logstash are you using? Have you check logstash logs? ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 3:44pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/3 "2016-11-22T15:44:43Z")
Here is the logstash logs, {:timestamp=\>"2016-11-22T15:43:44.577000+0000", :message=\>"Cannot get new connection from pool.", :class=\>"Elasticsearch::Transport::Transport::Error", :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:193:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:32:in `hosts'", "org/jruby/ext/timeout/Timeout.java:147:in`timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:76:in`reload\_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http\_client.rb:72:in `sniff!'", java/lib/logstash/output_delegator.rb:130:in`worker\_multi\_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/output\_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:293:in`output\_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:293:in`output\_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:224:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:193:in`start\_workers'"], :client\_config=\>{:hosts=\>["[http://search-oqelktest-srwthceqrt6hcadtissdrcjqa4.us-east-1.es.amazonaws.com/](http://search-oqelktest-srwthceqrt6hcadtissdrcjqa4.us-east-1.es.amazonaws.com/)"], :ssl=\>nil, :transport\_options=\>{:socket\_timeout=\>0, :request\_timeout=\>0, :proxy=\>nil, :ssl=\>{}}, :transport\_class=\>Elasticsearch::Transport::Transport::HTTP::Manticore, :logger=\>nil, :tracer=\>nil, :reload\_connections=\>false, :retry\_on\_failure=\>false, :reload\_on\_failure=\>false, :randomize\_hosts=\>false}, :level=\>:error} ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 3:48pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/4 "2016-11-22T15:48:59Z")
logstash 2.2.4 filebeat version 1.3.1 ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 22, 2016, 4:09pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/5 "2016-11-22T16:09:49Z")
this looks like a problem with logstash outputting to elasticsearch. Which logstash-input-beats plugin version is installed? Logstash has timeout on beats connections, in case of logstash pipeline being blocked. After a while logstash will close/reset connections. See `congestion_threshold` setting. Most recent plugin version removes the auto-closing if internal pipelines are closed. ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 4:42pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/6 "2016-11-22T16:42:59Z")
logstash-input-beats-2.2.7 is the version ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 4:45pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/7 "2016-11-22T16:45:59Z")
First of all, filebeat only not sending the events. 2016-11-22T16:43:04Z DBG Try to publish 932 events to logstash with window size 1 2016-11-22T16:43:04Z DBG close connection 2016-11-22T16:43:04Z DBG 0 events out of 932 events sent to logstash. Continue sending ... 2016-11-22T16:43:04Z INFO Error publishing events (retrying): EOF 2016-11-22T16:43:04Z INFO send fail 2016-11-22T16:43:04Z INFO backoff retry: 1m0s ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 5:07pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/8 "2016-11-22T17:07:44Z")
Also confused with below error in logstash. :message=\>"Cannot get new connection from pool.", :class=\>"Elasticsearch::Transport::Transport::Error" ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 5:40pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/9 "2016-11-22T17:40:24Z")
Missed one more information, we're using Elasticsearch service from AWS. ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 5:45pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/10 "2016-11-22T17:45:58Z")
@Robert_Firek , Could you help here? ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 22, 2016, 6:34pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/11 "2016-11-22T18:34:52Z")
As per @tomwj post [Elasitcsearch-ruby raises "Cannot get new connection from pool" error](https://discuss.elastic.co/t/elasitcsearch-ruby-raises-cannot-get-new-connection-from-pool-error/36252/12) I tried by removing sniffing =\> true. I'm able to insert data to elastic search by using nc command through tcp 5044 port with below config. [ec2-user@ip-10-3-1-13 conf.d]$ nc elk\_server\_ip.compute-1.amazonaws.com 5044 jashwanth dsjdskadjasdhskadf asfjkdsfhdsfkf fskjhasfkjfs asfjkfashsaf khfdslaf Config for testing: input { tcp { port =\> 5044 } } So, there is no issues with logstash and ES end now. Still able to see below error in filebeat end and suspecting the logs are not reaching logstash from filebeat. **2016-11-22T18:32:22Z DBG Try to publish 932 events to logstash with window size 1** **2016-11-22T18:32:22Z DBG close connection** **2016-11-22T18:32:22Z DBG 0 events out of 932 events sent to logstash. Continue sending ...** **2016-11-22T18:32:22Z INFO Error publishing events (retrying): EOF** **2016-11-22T18:32:22Z INFO send fail** **2016-11-22T18:32:22Z INFO backoff retry: 1m0s** ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 23, 2016, 9:58am UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/12 "2016-11-23T09:58:52Z")
Can you still check logstash logs? EOF means End Of File. You get this if connection has been closed by remote (connection closed by logstash). See my earlier post about setting the congestion threshold in logstash to not close connections. Filebeat is trying to send events, but logstash is not ACKing them. Due to internals in plugin version you're using, logstash seems to close the connection. Consider updating the beats input plugin. ---
**Author:** ![jashwanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jashwanth/32/100451_2.png) [@jashwanth](https://discuss.elastic.co/u/jashwanth)\ **Post date:** [November 23, 2016, 11:51am UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/13 "2016-11-23T11:51:32Z")
No, there is no issues with logstash server. I tried one more workaround. Launched a machine in Public subnet and installed filebeat, configured to send logs to same logstash server and started agent. This is working without any issues and i'm able to see the logs in Kibana. But facing issue with the filebeat agents which are installed in private subnet app servers. There is no port issues between these app servers and logstash server and still seeing same error in filebeat logs. No errors in logstash logs. ---
**Author:** ![Robert\_Firek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_firek/32/6353_2.png) [@Robert\_Firek](https://discuss.elastic.co/u/Robert_Firek)\ **Post date:** [November 23, 2016, 12:25pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/14 "2016-11-23T12:25:57Z")
Your problem looks very similar to my problem ([Elasitcsearch-ruby raises "Cannot get new connection from pool" error](https://discuss.elastic.co/t/elasitcsearch-ruby-raises-cannot-get-new-connection-from-pool-error/36252/9)), but maybe it is now problem in different plugin. I found my error by finding the exact place in the code where my exception was thrown (I would suggest to do the same). Next I modified the code to retrieve additional information about the connection (url, parameters, used libraries etc.). I compiled the code and deployed on the server. I repeated these steps few time until I found the reasons . In my case I found the code which depends on the API call which is not supported by AWS in the same way as Elasticsearch. It is a little bit primitive way of doing it, but only in such a way I was able to trace the root cause of the problem. Fortunately all my code was open sourced. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 23, 2016, 11:58pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/15 "2016-11-23T23:58:12Z")
I just found this bug report in logstash beats input plugin: [https://github.com/logstash-plugins/logstash-input-beats/issues/163](https://github.com/logstash-plugins/logstash-input-beats/issues/163) Seems some fix should be available in most recent plugin (version 3.1.10). Can you update the beats plugin and set `client_inactivity_timeout` in logstash to something very big (or 0 to disable logstash disconnecting clients) and see if this fixes your issue? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 21, 2016, 11:58pm UTC](https://discuss.elastic.co/t/filebeat-badly-stuck-with-connection-reset-by-peer-error/66878/16 "2016-12-21T23:58:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.