# Filebeat reports :Error publishing events (retrying): read tcp IPA-\>9IPB5044: i/o timeout, why?

**URL:** <https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398>\
**Category:** Beats\
**Created:** [June 14, 2017, 2:36pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398 "2017-06-14T14:36:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [June 14, 2017, 2:36pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/1 "2017-06-14T14:36:01Z")

</div>

HEllo,

I have updated my machine to filebeat 5 and now when trying to ship my logs I see that the shipping sometimes fails and I don't understand the reason.

The error is the following when enabling debug

```
2017-06-14T16:30:21+02:00 DBG Run prospector
2017-06-14T16:30:21+02:00 DBG Start next scan
2017-06-14T16:30:21+02:00 DBG Check file for harvesting: /var/log/user.log
2017-06-14T16:30:21+02:00 DBG Update existing file for harvesting: /var/log/user.log, offset: 7595
2017-06-14T16:30:21+02:00 DBG Harvester for file is still running: /var/log/user.log
2017-06-14T16:30:21+02:00 DBG Prospector states cleaned up. Before: 1, After: 1
2017-06-14T16:30:28+02:00 DBG handle error: read tcp IPA->IPB:5044: i/o timeout
2017-06-14T16:30:28+02:00 DBG 0 events out of 11 events sent to logstash. Continue sending
2017-06-14T16:30:28+02:00 DBG close connection
2017-06-14T16:30:28+02:00 DBG closing
2017-06-14T16:30:28+02:00 ERR Failed to publish events caused by: read tcp IPA->IPB:5044: i/o timeout
2017-06-14T16:30:28+02:00 INFO Error publishing events (retrying): read tcp IPA->IPB:5044: i/o timeout
2017-06-14T16:30:28+02:00 DBG close connection
2017-06-14T16:30:28+02:00 DBG send fail
2017-06-14T16:30:30+02:00 DBG connect
2017-06-14T16:30:30+02:00 DBG Try to publish 11 events to logstash with window size 2
2017-06-14T16:30:31+02:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=1390 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=703 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_but_not_acked_events=11
2017-06-14T16:30:31+02:00 DBG Run prospector

```

I have noticed that if I stop syslog daemon and filebeat, start first filebeat and then the syslog, filbeats sends out some logs properly but then happens as the above error.

My filebeat config is

```
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
     - /var/log/syslog.log
     - /var/log/user.log

  ignore_older: 24h

  scan_frequency: 10s 
   
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  include_lines: ['TTN-.*$']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  # level: debug
  # review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

filebeat.registry_file: /var/lib/filebeat/registry

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["IPB:5044"]
  bulk_max_size: 2048

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/mybeat
  name: mybeat.log
  keepfiles: 7
  rotateeverybytes: 10485760 # = 10MB
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 14, 2017, 11:29pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/2 "2017-06-14T23:29:30Z")

</div>

> [@ndarkness](#):
>
> ERR Failed to publish events caused by: read tcp IPA-\>IPB:5044: i/o timeout

That looks like a problem on the Logstash side. Please post your logstash config and the output of `sudo netstat -anp | grep 5044` on the Logstash host.

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [June 15, 2017, 7:55am UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/3 "2017-06-15T07:55:26Z")

</div>

> [@andrewkroh](#):
>
> That looks like a problem on the Logstash side. Please post your logstash config and the output of sudo netstat -anp | grep 5044 on the Logstash host.

My logstash config is this one

```
cat /etc/logstash/logstash.yml 
# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
# pipeline:
# batch:
# size: 125
# delay: 5
#
# Or as flat keys:
#
# pipeline.batch.size: 125
# pipeline.batch.delay: 5
#
# ------------ Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
path.data: /var/lib/logstash
#
# ------------ Pipeline Settings --------------
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many workers should be used per output plugin instance
#
# pipeline.output.workers: 1
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait before dispatching an undersized batch to filters+workers
# Value is in milliseconds.
#
# pipeline.batch.delay: 5
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
path.config: /etc/logstash/conf.d
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# Periodically check if the configuration has changed and reload the pipeline
# This can also be triggered manually through the SIGHUP signal
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
#
# config.reload.interval: 3
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
# queue.type: memory
#
# If using queue.type: persisted, the directory path where the data files will be stored.
# Default is path.data/queue
#
# path.queue:
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 250mb
#
# queue.page_capacity: 250mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
# queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []

```

Hitting `sudo netstat -anp | grep 5044` gives

```
tcp 0 0 95.85.38.21:54573 95.85.38.21:5044 FIN_WAIT2 -               
tcp6 0 0 :::5044 :::* LISTEN 13904/java      
tcp6 54 0 95.85.38.21:5044 146.185.166.84:37976 CLOSE_WAIT 13904/java      
tcp6 3916 0 95.85.38.21:5044 146.185.166.84:38149 CLOSE_WAIT 13904/java      
tcp6 54 0 95.85.38.21:5044 95.85.38.21:57498 CLOSE_WAIT 13904/java      
tcp6 54 0 95.85.38.21:5044 95.85.38.21:54573 CLOSE_WAIT 13904/java      
tcp6 54 0 95.85.38.21:5044 146.185.166.84:38114 CLOSE_WAIT 13904/java      
tcp6 54 0 95.85.38.21:5044 146.185.166.84:41271 CLOSE_WAIT 13904/java

```

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [June 15, 2017, 8:17am UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/4 "2017-06-15T08:17:08Z")

</div>

Checking the logs of logstash show these errors

> ```
> {:timestamp=>"2017-06-15T06:51:23.722000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200/\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :client_config=>{:hosts=>["http://localhost:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :error_message=>"Cannot get new connection from pool.", :error_class=>"Elasticsearch::Transport::Transport::Error", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:193:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:125:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/bulk.rb:87:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:162:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/output_delegator.rb:119:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/output_delegator.rb:118:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/output_delegator.rb:65:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:290:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:290:in `output_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:221:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:190:in `start_workers'"], :level=>:error}
> {:timestamp=>"2017-06-15T06:51:23.722000+0200", :message=>"Cannot get new connection from pool.", :class=>"Elasticsearch::Transport::Transport::Error", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:193:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:125:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/bulk.rb:87:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:162:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/output_delegator.rb:119:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/output_delegator.rb:118:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/output_delegator.rb:65:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:290:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:290:in `output_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:221:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:190:in `start_workers'"], :level=>:warn}
> 
> ```

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [June 15, 2017, 12:47pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/5 "2017-06-15T12:47:52Z")

</div>

@andrewkroh I have reinstalled logstash and now it the port is not used anymore...

```
sudo netstat -anp | grep 5044
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 15, 2017, 2:14pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/6 "2017-06-15T14:14:01Z")

</div>

What config do you have in `/etc/logstash/conf.d`?

---

<div class="post-metadata">

**Author:** ![debashree13](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@debashree13](https://discuss.elastic.co/u/debashree13)\
**Post date:** [June 15, 2017, 7:48pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/7 "2017-06-15T19:48:05Z")

</div>

> [@ndarkness](#):
>
> sudo netstat -anp | grep 5044

What is the update, are we able to solve this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 2:36pm UTC](https://discuss.elastic.co/t/filebeat-reports-error-publishing-events-retrying-read-tcp-ipa-9ipb5044-i-o-timeout-why/89398/8 "2017-07-05T14:36:26Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.