# Lumberjack output plugin posts duplicate entries when there is a connection error **URL:** \ **Category:** Logstash\ **Created:** [September 24, 2018, 4:33am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615 "2018-09-24T04:33:11Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![manz](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@manz](https://discuss.elastic.co/u/manz)\ **Post date:** [September 24, 2018, 4:33am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615/1 "2018-09-24T04:33:11Z")
Duplicate entries are posted when the following error happens. * * * Client write error, trying connect {:e=\>#\, :backtrace=\>["org/jruby/ext/openssl/SSLSocket.java:950:in `syswrite'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:107:in`send\_window\_size'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:127:in `write_sync'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in`write'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in `flush'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in`block in buffer\_flush'", "org/jruby/RubyHash.java:1343:in `each'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in`buffer\_flush'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in`block in register'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in `encode'", "/root/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in`receive'", "/root/logstash-6.4.0/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1734:in`each'", "/root/logstash-6.4.0/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in`multi\_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/root/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:372:in`block in output\_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/root/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:371:in`output\_batch'", "/root/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:323:in `worker_loop'", "/root/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:285:in`block in start\_workers'"]} * * * Any idea why this error happens? I am using logstah 6.3.2 in both server and client side. and my yml file looks like... ########################## #[https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html](https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html) input { file { path =\> "/home/abcdef/kibana\_in/in.json" start\_position =\> "beginning" sincedb\_path =\> "/home/abcdef/sincedb\_path.txt" codec =\> "json" } } filter { # split { } } output { lumberjack { codec =\> json hosts =\> "[abcd.abcd.com](http://abcd.abcd.com)" ssl\_certificate =\> "/home/abcdef/lumberjack.cert" port =\> 31333 } file { path =\> "/home/abcdef/file.out.txt" } } ##################################### Please help !!! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 26, 2018, 4:44am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615/2 "2018-09-26T04:44:36Z")
If the connection breaks after the lumberjack output has sent the data but before it has received the acknowledgement from the peer that data payload will get sent again. There's nothing to do about that. ---
**Author:** ![manz](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@manz](https://discuss.elastic.co/u/manz)\ **Post date:** [September 27, 2018, 4:58am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615/3 "2018-09-27T04:58:33Z")
Hi, Any idea why this connection error happens frequently? Is there any suggestion to avoid duplicate entries if the error is unavoidable? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 27, 2018, 5:48am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615/4 "2018-09-27T05:48:05Z")
Most components in the Elastic stack offer an at-least-once delivery guarantee with respect to network errors like this, so if issues in your network can not be avoided, duplicates are difficult to avoid throughout the pipeline. This is however instead often handled at the destination system. When data e.g. is sent to Elasticsearch, one can [as outlined in this blog post](https://www.elastic.co/blog/logstash-lessons-handling-duplicates) specify an external document ID. This makes Elasticsearch index the first document, but apply all duplicates having the same document IDs as updates thereby replacing the first document instead of adding them as separate documents. ---
**Author:** ![manz](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@manz](https://discuss.elastic.co/u/manz)\ **Post date:** [October 3, 2018, 6:32am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615/5 "2018-10-03T06:32:12Z")
Thanks a lot ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 31, 2018, 6:32am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615/6 "2018-10-31T06:32:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.