# Lumberjack error: Client write error, trying connect (..syswrite, write\_sync, multi\_receive, output\_batch, worker\_loop, start\_workers)

**URL:** <https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557>\
**Category:** Logstash\
**Created:** [September 30, 2020, 6:59pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557 "2020-09-30T18:59:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stanvv](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Stanvv](https://discuss.elastic.co/u/Stanvv)\
**Post date:** [September 30, 2020, 6:59pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557/1 "2020-09-30T18:59:20Z")

</div>

When trying to forward data via the lumberjack protocol, we get the following error:

> **See full error**
>
> [2020-09-30T20:23:04,014][ERROR][logstash.outputs.lumberjack] Client write error, trying connect {:e=\>#\<IOError: Connection reset by peer\>, :backtrace=\>["org/jruby/ext/openssl/SSLSocket.java:950:in `syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in `send\_window\_size'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in `write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in `flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer\_flush'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer\_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi\_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in `block in output\_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in `output\_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start\_workers'"]}

Since we have quite a complex setup (it's a PoC) please find the schematic view below.

> **Schematic view**
>
> ![logstash_error](https://us1.discourse-cdn.com/elastic/original/3X/9/3/93bdfae69accc3f59b078bf759ee3e8233ccf7d4.png)

More info:

- The error occurs on Logstash 1, Logstash 2 and Logstash 3.
- Please note the different versions of Logstash which make it more complex
- A beats node with output straight towards Logstash 3 does not give this error
- The error is quite similar to [this](https://discuss.elastic.co/t/logstash-fatal-error-when-cannot-connect-to-elasticsearch-v6-2-3/129377) topic, therefore I tried filters [uuid](https://www.elastic.co/guide/en/logstash/7.9/plugins-filters-uuid.html) and [fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) in different setups on Logstash 1, 2 and 3
- As output on Logstash 1, 2 and 3 we use a fairly simple lumberjack configuration:

> **Lumberjack output Logstash 1 and 2**
>
> ```
> output {
> lumberjack {
> codec => json
> hosts => ["dns_name_logstash_server_3"]
> ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
> port => 6000
> }
> }
> 
> ```

> **Lumberjack output Logstash 3**
>
> ```
> output {
> lumberjack {
> codec => json
> hosts => ["address_to_loadbalancer_logstash4_and5"] 
> ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
> port => 6000
> }
> }
> 
> ```

Any thoughts?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 30, 2020, 7:38pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557/2 "2020-09-30T19:38:29Z")

</div>

> [@Stanvv](#):
>
> Any thoughts?

If logstash1 logs 'Connection reset by peer' I would expect logstash3 to log a reason why it dropped the connection.

What input are you connecting the lumberjack output to?

Oh, and even with 6.6.2 in the picture, I think lumberjack stopped being the preferred way of doing logstash-to-logstash communication. tcp/tcp or http/http are other options.

---

<div class="post-metadata">

**Author:** ![Stanvv](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Stanvv](https://discuss.elastic.co/u/Stanvv)\
**Post date:** [October 1, 2020, 6:28am UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557/3 "2020-10-01T06:28:21Z")

</div>

For Logstash1, 2 and 3 the input is beats, see below (to make it more readable, I skipped the pipelines.yml file)

> **Configuration Logstash 1 and 2**
>
> ```
> input { beats { port => 5044 } }
> 
> filter {
> mutate {
> add_tag => ["my_tags"]
> add_field => { "agent_host" => "%{[beat][hostname]}" }
> }
> if ! [host][name] { mutate { rename => { "[host]" => "[host][name]" } } }
> dns {
> resolve => ["agent_host"]
> action => "replace"
> }
> dns {
> reverse => ["agent_host"]
> action => "replace"
> }
> mutate {
> replace => {"[host][name]" => "%{agent_host}"}
> remove_field => ["agent_host"]
> }
> }
> 
> output {
> lumberjack {
> codec => json
> hosts => ["dns_name_logstash_server_3"]
> ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
> port => 6000
> }
> }
> 
> ```

> **Configuration Logstash 3**
>
> ```
> input {
> beats {
> port => 6000
> ssl => true
> ssl_key => '/etc/logstash/certs/lumberjack.key'
> ssl_certificate => '/etc/logstash/certs/lumberjack.crt'
> codec => json
> }
> }
> filter {
> mutate {
> add_tag => ["tag_name"]
> }
> uuid {
> target => "uuid"
> overwrite => true
> }
> }
> output {
> lumberjack {
> codec => json
> hosts => ["address_to_loadbalancer_logstash4_and5"] 
> ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
> port => 6000
> }
> }
> 
> ```

I see the errors occur on Logstash 1 and 2, as well as on 3 (mostly at the same time):

> **Error Logstash2**
>
> `[2020-10-01T08:20:16,117][ERROR][logstash.outputs.lumberjack] Client write error, trying connect {:e=>#<IOError: Connection reset by peer>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:950:in `syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in `send_window_size'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in `write\_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in `flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer\_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi\_receive'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi\_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi\_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `worker\_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}`

> **Error Logstash3**
>
> `[2020-10-01T08:22:37,137][ERROR][logstash.outputs.lumberjack][pipeline-data] Client write error, trying connect {:e=>#<IOError: Connection reset by peer>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:950:in `syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in `send_window_size'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in `write\_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in `flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1428:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer\_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in `encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:31:in `block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:45:in `time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:44:in `time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:30:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi\_receive'", "org/jruby/RubyArray.java:1814:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi\_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi\_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:262:in `block in start_workers'"]}`

> [@Badger](#):
>
> Oh, and even with 6.6.2 in the picture, I think lumberjack stopped being the preferred way of doing logstash-to-logstash communication. tcp/tcp or http/http are other options.

Ah, interesting, do you have any documentation on that? I cannot find it on the [Logstash-to-logstash page](https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 1, 2020, 2:52pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557/4 "2020-10-01T14:52:38Z")

</div>

OK, so logstash3 does not log an error when, or shortly before, logstash2 get the error writing to it. I do not know what to suggest.

You are right that lumberjack to beats is Elastic's documented way of connecting logstash to logstash. I must be mis-remembering it.

---

<div class="post-metadata">

**Author:** ![Stanvv](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Stanvv](https://discuss.elastic.co/u/Stanvv)\
**Post date:** [October 2, 2020, 1:21pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557/5 "2020-10-02T13:21:09Z")

</div>

For now I switched to lumberjack input on Logstash3 and so far it seems to be working fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 30, 2020, 1:21pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557/6 "2020-10-30T13:21:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.