# Filebeat connection refused eof

**URL:** <https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 9, 2016, 3:26pm UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914 "2016-03-09T15:26:33Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![cernicb](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@cernicb](https://discuss.elastic.co/u/cernicb)\
**Post date:** [March 9, 2016, 3:26pm UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/1 "2016-03-09T15:26:33Z")

</div>

We use filebeat -\> logstash to transport logs to graylog. As soon as filebeat logs:

> Error publishing events (retrying): EOF

it is followed by:

> Connecting error publishing events (retrying): dial tcp y.y.y.y:5043: getsockopt: connection refused

and connection to logstash is lost, and status check on logstash returns "not running". Logstash logs something like:

> {:timestamp=\>"2016-03-08T16:08:27.658000+0100", :message=\>"Beats Input: Remote connection closed", :peer=\>"x.x.x.x", :exception=\>#, :level=\>:warn}

x.x.x.x and y.y.y.y ping each other.

After that point, there are no incoming logs on graylog. Once we start logstash again, everything is ok until new EOF arises.

We use logstash-2.2.2-1.noarch with beat plugin 2.1.4, running on CentOS release 6.6, and filebeat version 1.1.1 (amd64) on CentOS release 6.4 (Final).

Output configuration on filebeat is:  
`logstash: # The Logstash hosts hosts: ["y.y.y.y:5043"]`  
with one worker and no tls.

Input configuration on logstash is:  
`input { beats { port => 5043 } }`

Any help appreciated.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 10, 2016, 11:31am UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/2 "2016-03-10T11:31:52Z")

</div>

There are two open and one closed issue related to this in logstash-input-beats: [https://github.com/logstash-plugins/logstash-input-beats/issues?utf8=✓&q=is%3Aissue++Remote+connection](https://github.com/logstash-plugins/logstash-input-beats/issues?utf8=%E2%9C%93&q=is%3Aissue++Remote+connection) Please check if one of these matches your pattern.

---

<div class="post-metadata">

**Author:** ![cernicb](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@cernicb](https://discuss.elastic.co/u/cernicb)\
**Post date:** [March 11, 2016, 3:08pm UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/3 "2016-03-11T15:08:33Z")

</div>

Thanks ruflin,

I checked these issues, but didn't find any solution. We already use input beats plugin 2.1.4. We have plain tcp between filebeat and logstash, no ssl, and no proxy. We tried to downgrade to logstash 2.1.9 as suggested in closed issue, but have found only 2.1.3, with no results after downgrade. I turned debug log level on filebeat, and here's what we noticed:

> 2016-03-11T15:22:43+01:00 DBG full line read  
> 2016-03-11T15:22:43+01:00 DBG End of file reached: /usr/local/mpg/log/mpg.log; Backoff now.  
> 2016-03-11T15:22:43+01:00 DBG connect  
> 2016-03-11T15:22:43+01:00 INFO Connecting error publishing events (retrying): dial tcp x.x.x.x:5043: getsockopt: connection refused  
> 2016-03-11T15:22:43+01:00 INFO send fail  
> 2016-03-11T15:22:43+01:00 INFO backoff retry: 8s  
> 2016-03-11T15:22:44+01:00 DBG Flushing spooler because of timeout. Events flushed: 654  
> 2016-03-11T15:22:44+01:00 DBG full line read

There was no connection to logstash for only short period of time (reason unknown - EOF?), and after backoff, new lines are found. Logstash is running, nothing wrong.

> 2016-03-11T15:22:44+01:00 DBG End of file reached: /usr/local/mpg/log/mpg.log; Backoff now.  
> 2016-03-11T15:22:45+01:00 DBG full line read

Yet another EOF, but new line is found immediately after. Logstash is running.

> 2016-03-11T15:22:50+01:00 DBG full line read  
> 2016-03-11T15:22:51+01:00 DBG connect  
> 2016-03-11T15:22:51+01:00 INFO Connecting error publishing events (retrying): dial tcp 10.50.110.20:5043: getsockopt: connection refused  
> 2016-03-11T15:22:51+01:00 INFO send fail  
> 2016-03-11T15:22:51+01:00 INFO backoff retry: 16s  
> 2016-03-11T15:22:59+01:00 DBG Start next scan  
> 2016-03-11T15:22:59+01:00 DBG scan path /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:22:59+01:00 DBG Check file for harvesting: /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:22:59+01:00 DBG Update existing file for harvesting: /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:22:59+01:00 DBG Not harvesting, file didn't change: /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:23:07+01:00 DBG connect  
> 2016-03-11T15:23:07+01:00 INFO Connecting error publishing events (retrying): dial tcp 10.50.110.20:5043: getsockopt: connection refused  
> 2016-03-11T15:23:07+01:00 INFO send fail  
> 2016-03-11T15:23:07+01:00 INFO backoff retry: 32s  
> 2016-03-11T15:23:09+01:00 DBG Start next scan  
> 2016-03-11T15:23:09+01:00 DBG scan path /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:23:09+01:00 DBG Check file for harvesting: /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:23:09+01:00 DBG Update existing file for harvesting: /usr/local/mpg/log/mpg.log  
> 2016-03-11T15:23:09+01:00 DBG Not harvesting, file didn't change: /usr/local/mpg/log/mpg.log

No connection to logstash once again, but this time no harvesting as well, file didn't change (EOF again?). After two backoffs, file still didn't change. At this point, a checked status of logstash returned 'not running', it was shutdown.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 14, 2016, 8:06am UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/4 "2016-03-14T08:06:48Z")

</div>

If I understand you right, in the end Logstash was running anymore, means it shutdown / crashed without doing anything on the Logstash side?

---

<div class="post-metadata">

**Author:** ![cernicb](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@cernicb](https://discuss.elastic.co/u/cernicb)\
**Post date:** [March 14, 2016, 8:27am UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/5 "2016-03-14T08:27:09Z")

</div>

That's right, logstash shutdown, and the only thing in the log indicating any sign of break down is:

> {:timestamp=\>"2016-03-08T16:08:27.658000+0100", :message=\>"Beats Input: Remote connection closed", :peer=\>"x.x.x.x", :exception=\>#, :level=\>:warn}

---

<div class="post-metadata">

**Author:** ![cernicb](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@cernicb](https://discuss.elastic.co/u/cernicb)\
**Post date:** [March 17, 2016, 9:27am UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/6 "2016-03-17T09:27:47Z")

</div>

ruflin,  
Thanks for the effort, problem was on our side with linux user privileges. Administrator solved the mystery. Filebeat rocks now 🙂

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 18, 2016, 8:37am UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/7 "2016-03-18T08:37:11Z")

</div>

@cernicb Good to hear. Thanks for also posting the "solution".

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:54pm UTC](https://discuss.elastic.co/t/filebeat-connection-refused-eof/43914/8 "2017-07-05T21:54:31Z")

</div>


