# Lumberjack output "Broken Pipe"

**URL:** https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936
**Category:** Logstash
**Created:** [April 2, 2019, 8:02am UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936 "2019-04-02T08:02:21Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![gruselglatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gruselglatz/32/42675_2.png) [@gruselglatz](https://discuss.elastic.co/u/gruselglatz)
#### Post date: [April 2, 2019, 8:02am UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/1 "2019-04-02T08:02:22Z")

</div>

Hi,

i've tried the lumberjack output to utilize the graylog beats input, but it doesn't get a connection.

IDK if this is a problem on the logstash or graylog side. Does someone have tried to connect lumberjack to the beats input and got it working?

I get this messages on graylog:

```
2019-04-01T11:28:00.230+02:00 ERROR [AbstractTcpTransport] Error in Input [Beats/5c6acc71a0303c17f682ba78] (channel [id: 0xa2af5fea, L:/172.21.2.63:5044 ! R:/172.21.2.99:46936]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 49)
2019-04-01T11:28:00.231+02:00 ERROR [AbstractTcpTransport] Error in Input [Beats/5c6acc71a0303c17f682ba78] (channel [id: 0xa2af5fea, L:/172.21.2.63:5044 ! R:/172.21.2.99:46936]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 87)
2019-04-01T11:28:49.442+02:00 ERROR [AbstractTcpTransport] Error in Input [Beats/5c6acc71a0303c17f682ba78] (channel [id: 0x8e05800f, L:/172.21.2.63:5044 ! R:/172.21.2.99:46940]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 10)

```

this on logstash:

```
Client write error, trying connect {:e=>#<IOError: Broken pipe>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:857:in `sysread'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-openssl-0.10.2-java/lib/jopenssl23/openssl/buffering.rb:57:in `fill_rbuff'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-openssl-0.10.2-java/lib/jopenssl23/openssl/buffering.rb:98:in `read'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:157:in `read_version_and_type'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:145:in `ack'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:134:in `write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in `flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:40:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in `block in output_batch'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}

```

thit is the openssl output:

```
openssl s_client -connect graylog.domain.com:5044 -CAfile ca-pem.crt
CONNECTED(00000003)
depth=1 DC = com, DC = domain, CN = CA
verify return:1
depth=0 CN = graylog.domain.com
verify return:1
---
Certificate chain
 0 s:/CN=graylog.domain.com
   i:/DC=com/DC=domain/CN=CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/CN=graylog.domain.com
issuer=/DC=com/DC=domain/CN=CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1972 bytes and written 269 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol : TLSv1.2
    Cipher : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 485AB5E76E1FBECBDAF7027B180CD953A24505BFE0BF1C517BDBB3947ECCEB08
    Session-ID-ctx:
    Master-Key: 8E4A92AEFF1BB36ADF3D22331273F51D1D21AA802A988C21B787EF8CF46A7CA5BE7A3AC6837DED7869D5DF0A1CA071CE
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1554110924
    Timeout : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

```

this is the input:

```
bind_address: 0.0.0.0
no_beats_prefix: false
number_worker_threads: 4
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: /home/rene/graylog.domain.com.crt
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: true
tls_key_file: /home/rene/graylog.domain.com.rsa
tls_key_password: ********

```

this is the logstash config:

```
input {
  beats {
    port => 5050
    ssl => true
    ssl_certificate => "/home/logstash/certs/agg_hag1.crt"
    ssl_key => "/home/logstash/certs/agg_hag1-des-v1.pem"
    ssl_key_passphrase => "PWD"
    ssl_verify_mode => none
    tls_min_version => 1.2
  }
}

output {
  lumberjack {
    id => "internal_POC"
    enable_metric => true
    hosts => "graylog.domain.com"
    port => 5044
    ssl_certificate => "/home/logstash/certs/ca-pem.crt"
  }
}
```

---

<div class="post-metadata">

### Author: ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 2, 2019, 4:09pm UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/2 "2019-04-02T16:09:28Z")

</div>

Reading the Graylog Sidecar docs I get the impression that they have reverse engineered a Beats compatible input as a target for Beats traffic.

The lumberjack output uses the Lumberjack protocol. I know the Logstash beats input understands the Lumberjack protocol but I'm not sure whether the Graylog Sidecar Beats input does - I checked the source and I can't see any ref to Lumberjack the protocol, only the Golang log-rotation package of the same name.

---

<div class="post-metadata">

### Author: ![gruselglatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gruselglatz/32/42675_2.png) [@gruselglatz](https://discuss.elastic.co/u/gruselglatz)
#### Post date: [April 3, 2019, 8:55am UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/3 "2019-04-03T08:55:00Z")

</div>

Hi @guyboertje!

Thank you for you time! I've been using the Beats input plugin in Graylog itself and i found this code: [https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog/plugins/beats/BeatsFrameDecoder.java](https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog/plugins/beats/BeatsFrameDecoder.java)

So you say, they arent compatible with lumberjack? Then I have to switch to GELF output.

---

<div class="post-metadata">

### Author: ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 3, 2019, 9:00am UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/4 "2019-04-03T09:00:02Z")

</div>

Ahh, I did not find that code - I only looked in Sidecar.

However, the last change was made one year ago. Perhaps, we made a change to the Lumberjack protocol since then?

I don't think any one here can help much more than this. Perhaps you should try the Graylog folks?

---

<div class="post-metadata">

### Author: ![gruselglatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gruselglatz/32/42675_2.png) [@gruselglatz](https://discuss.elastic.co/u/gruselglatz)
#### Post date: [April 3, 2019, 9:02am UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/5 "2019-04-03T09:02:23Z")

</div>

Thank you @guyboertje ! I've tried it in the graylog forums, but they say they are compatible ([https://community.graylog.org/t/filebeat-logstash-graylog-all-ssl-secured/9668/9](https://community.graylog.org/t/filebeat-logstash-graylog-all-ssl-secured/9668/9)). But idk, maybe it's less stress and pain when i switch to gelf for the last mile 🙄

Thank you anyway 😃

---

<div class="post-metadata">

### Author: ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 3, 2019, 11:43am UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/6 "2019-04-03T11:43:19Z")

</div>

There will always be a risk that we change the Lumberjack protocol out of sync with the Graylog implementation - therefore it is better in the long run to use more stable technologies.

Elastic obviously controls the Lumberjack protocol and will ensure that our products using it remain in sync but that can't be guaranteed for 3rd party implementations.

---

<div class="post-metadata">

### Author: ![gruselglatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gruselglatz/32/42675_2.png) [@gruselglatz](https://discuss.elastic.co/u/gruselglatz)
#### Post date: [April 3, 2019, 1:58pm UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/7 "2019-04-03T13:58:17Z")

</div>

Hi again, @guyboertje !

Is it officialy supported to do lumberjack -\> beats? I've followed this instruction, but i cant connect bcs of strange ssl errors: [https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html](https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html)

i get sometimes PEER\_DID\_NOT\_RETURN\_A\_CERTIFICATE or SSLV3\_ALERT\_CERTIFICATE\_UNKNOWN

ERRORS:

```
[2019-04-03T15:14:47,320][INFO][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5055, remote: 172.21.2.97:49628] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
[2019-04-03T15:14:47,323][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1140) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1101) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1169) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1212) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1211) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        ... 16 more

```

my configs:  
upstream:

```
output {
  lumberjack {
    hosts => "agg_hag1.domain.com"
    port => 5055
    ssl_certificate => "/home/logstash/certs/agg_hag1.crt"
    codec => "json"
  }
}

```

downstream:

```
input {
  beats {
    port => 5055
    codec => "json"
    ssl => true
    ssl_certificate => "/home/logstash/certs/agg_hag1.crt"
    ssl_certificate_authorities => "/home/logstash/certs/domain-ca-pem.crt"
    ssl_key => "/home/logstash/certs/agg_hag1-des-v1.pem"
    ssl_key_passphrase => "PWD"
# ssl_verify_mode => none
# tls_min_version => 1.2
  }
}
```

---

<div class="post-metadata">

### Author: ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 3, 2019, 2:03pm UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/8 "2019-04-03T14:03:05Z")

</div>

Would you mind creating a new discussion thread as this one is marked as solved and the title is not LS -\> LS related?  
So other folks can help or get a solution for LS -\> LS.

---

<div class="post-metadata">

### Author: ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 3, 2019, 2:12pm UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/9 "2019-04-03T14:12:52Z")

</div>

Or we can work on the other one you commented on [Logstash Lumberjack Cert](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/4?u=guyboertje)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 1, 2019, 2:12pm UTC](https://discuss.elastic.co/t/lumberjack-output-broken-pipe/174936/10 "2019-05-01T14:12:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
