Lumberjack error: Client write error, trying connect (..syswrite, write_sync, multi_receive, output_batch, worker_loop, start_workers)

When trying to forward data via the lumberjack protocol, we get the following error:

See full error

[2020-09-30T20:23:04,014][ERROR][logstash.outputs.lumberjack] Client write error, trying connect {:e=>#<IOError: Connection reset by peer>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:950:in syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in send_window_size'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in write'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in block in buffer_flush'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in block in multi_receive'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in block in start_workers'"]}

Since we have quite a complex setup (it's a PoC) please find the schematic view below.

Schematic view

More info:

  • The error occurs on Logstash 1, Logstash 2 and Logstash 3.
  • Please note the different versions of Logstash which make it more complex
  • A beats node with output straight towards Logstash 3 does not give this error
  • The error is quite similar to this topic, therefore I tried filters uuid and fingerprint in different setups on Logstash 1, 2 and 3
  • As output on Logstash 1, 2 and 3 we use a fairly simple lumberjack configuration:
Lumberjack output Logstash 1 and 2
output {
    lumberjack {
        codec => json
        hosts => ["dns_name_logstash_server_3"]
        ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
        port => 6000
    }
  }
Lumberjack output Logstash 3
output {
    lumberjack {
        codec => json
        hosts => ["address_to_loadbalancer_logstash4_and5"] 
        ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
        port => 6000
    }
  }

Any thoughts?

If logstash1 logs 'Connection reset by peer' I would expect logstash3 to log a reason why it dropped the connection.

What input are you connecting the lumberjack output to?

Oh, and even with 6.6.2 in the picture, I think lumberjack stopped being the preferred way of doing logstash-to-logstash communication. tcp/tcp or http/http are other options.

For Logstash1, 2 and 3 the input is beats, see below (to make it more readable, I skipped the pipelines.yml file)

Configuration Logstash 1 and 2
input { beats { port => 5044 } }

filter {
    mutate {
        add_tag => [ "my_tags" ]
        add_field => { "agent_host" => "%{[beat][hostname]}" }
      }
    if ! [host][name] { mutate { rename => { "[host]" => "[host][name]" } } }
    dns {
        resolve => ["agent_host"]
        action => "replace"
      }
    dns {
        reverse => ["agent_host"]
        action => "replace"
      }
    mutate {
        replace => {"[host][name]" => "%{agent_host}"}
        remove_field => ["agent_host"]
      }
}

output {
    lumberjack {
        codec => json
        hosts => ["dns_name_logstash_server_3"]
        ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
        port => 6000
    }
  }
Configuration Logstash 3
  input {
      beats {
              port => 6000
              ssl => true
              ssl_key => '/etc/logstash/certs/lumberjack.key'
              ssl_certificate => '/etc/logstash/certs/lumberjack.crt'
              codec => json
                   }
          }
filter {
    mutate {
        add_tag => [ "tag_name" ]
    }
    uuid {
        target    => "uuid"
        overwrite => true
      }
}
output {
    lumberjack {
        codec => json
        hosts => ["address_to_loadbalancer_logstash4_and5"] 
        ssl_certificate => "/etc/logstash/certs/lumberjack.crt"
        port => 6000
    }
  }

I see the errors occur on Logstash 1 and 2, as well as on 3 (mostly at the same time):

Error Logstash2

[2020-10-01T08:20:16,117][ERROR][logstash.outputs.lumberjack] Client write error, trying connect {:e=>#<IOError: Connection reset by peer>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:950:in syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in send_window_size'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in write'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in block in buffer_flush'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in block in multi_receive'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in block in start_workers'"]}

Error Logstash3

[2020-10-01T08:22:37,137][ERROR][logstash.outputs.lumberjack][pipeline-data] Client write error, trying connect {:e=>#<IOError: Connection reset by peer>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:950:in syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in send_window_size'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in write'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in block in buffer_flush'", "org/jruby/RubyHash.java:1428:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:31:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:45:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:44:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:30:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in block in multi_receive'", "org/jruby/RubyArray.java:1814:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:262:in block in start_workers'"]}

Ah, interesting, do you have any documentation on that? I cannot find it on the Logstash-to-logstash page

OK, so logstash3 does not log an error when, or shortly before, logstash2 get the error writing to it. I do not know what to suggest.

You are right that lumberjack to beats is Elastic's documented way of connecting logstash to logstash. I must be mis-remembering it.

For now I switched to lumberjack input on Logstash3 and so far it seems to be working fine.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.