# Filebeats 6.3 tcp client crash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 29, 2018, 9:49pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759 "2018-05-29T21:49:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![emrith](https://avatars.discourse-cdn.com/v4/letter/e/76d3ee/32.png) [@emrith](https://discuss.elastic.co/u/emrith)\ **Post date:** [May 29, 2018, 9:49pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/1 "2018-05-29T21:49:59Z")
I'm running filebeat version 6.3.0 (amd64), libbeat 6.3.0 [489ad50276d34fd597cb31faa644aba1ecf47574 built 2018-05-11 19:02:09 +0000 UTC] after a week of running we get this in the log 2018-05-29T10:55:39.577-0600 ERROR sync/waitgroup.go:73 recovering from a tcp client crash. Recovering, but please report this. {"panic": "sync: negative WaitGroup counter", "stack": "[github.com/elastic/beats/libbeat/logp.Recover\n\t/home/jason/go/src/github.com/elastic/beats/libbeat/logp/global.go:88\nruntime.call32\n\t/usr/lib/go-1.10/src/runtime/asm\_amd64.s:573\nruntime.gopanic\n\t/usr/lib/go-1.10/src/runtime/panic.go:502\nsync.(\*WaitGroup](http://github.com/elastic/beats/libbeat/logp.Recover%5Cn%5Ct/home/jason/go/src/github.com/elastic/beats/libbeat/logp/global.go:88%5Cnruntime.call32%5Cn%5Ct/usr/lib/go-1.10/src/runtime/asm_amd64.s:573%5Cnruntime.gopanic%5Cn%5Ct/usr/lib/go-1.10/src/runtime/panic.go:502%5Cnsync.(*WaitGroup)).Add\n\t/usr/lib/go-1.10/src/sync/waitgroup.go:73\[ngithub.com/elastic/beats/filebeat/beater.(\*eventCounter](http://ngithub.com/elastic/beats/filebeat/beater.(*eventCounter)).Add\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/beater/channels.go:61\[ngithub.com/elastic/beats/filebeat/channel.(\*outlet](http://ngithub.com/elastic/beats/filebeat/channel.(*outlet)).OnEvent\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/channel/outlet.go:43\[ngithub.com/elastic/beats/filebeat/harvester.(\*Forwarder](http://ngithub.com/elastic/beats/filebeat/harvester.(*Forwarder)).Send\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:33\[ngithub.com/elastic/beats/filebeat/input/tcp.NewInput.func1\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/input/tcp/input.go:59\ngithub.com/elastic/beats/filebeat/inputsource/tcp.(\*client](http://ngithub.com/elastic/beats/filebeat/input/tcp.NewInput.func1%5Cn%5Ct/home/jason/go/src/github.com/elastic/beats/filebeat/input/tcp/input.go:59%5Cngithub.com/elastic/beats/filebeat/inputsource/tcp.(*client)).handle\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/inputsource/tcp/client.go:71\[ngithub.com/elastic/beats/filebeat/inputsource/tcp.(\*Server](http://ngithub.com/elastic/beats/filebeat/inputsource/tcp.(*Server)).run.func1\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/inputsource/tcp/server.go:99"} 2018-05-29T10:55:39.637-0600 ERROR sync/waitgroup.go:73 recovering from a tcp client crash. Recovering, but please report this. {"panic": "sync: negative WaitGroup counter", "stack": "[github.com/elastic/beats/libbeat/logp.Recover\n\t/home/jason/go/src/github.com/elastic/beats/libbeat/logp/global.go:88\nruntime.call32\n\t/usr/lib/go-1.10/src/runtime/asm\_amd64.s:573\nruntime.gopanic\n\t/usr/lib/go-1.10/src/runtime/panic.go:502\nsync.(\*WaitGroup](http://github.com/elastic/beats/libbeat/logp.Recover%5Cn%5Ct/home/jason/go/src/github.com/elastic/beats/libbeat/logp/global.go:88%5Cnruntime.call32%5Cn%5Ct/usr/lib/go-1.10/src/runtime/asm_amd64.s:573%5Cnruntime.gopanic%5Cn%5Ct/usr/lib/go-1.10/src/runtime/panic.go:502%5Cnsync.(*WaitGroup)).Add\n\t/usr/lib/go-1.10/src/sync/waitgroup.go:73\[ngithub.com/elastic/beats/filebeat/beater.(\*eventCounter](http://ngithub.com/elastic/beats/filebeat/beater.(*eventCounter)).Add\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/beater/channels.go:61\[ngithub.com/elastic/beats/filebeat/channel.(\*outlet](http://ngithub.com/elastic/beats/filebeat/channel.(*outlet)).OnEvent\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/channel/outlet.go:43\[ngithub.com/elastic/beats/filebeat/harvester.(\*Forwarder](http://ngithub.com/elastic/beats/filebeat/harvester.(*Forwarder)).Send\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:33\[ngithub.com/elastic/beats/filebeat/input/tcp.NewInput.func1\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/input/tcp/input.go:59\ngithub.com/elastic/beats/filebeat/inputsource/tcp.(\*client](http://ngithub.com/elastic/beats/filebeat/input/tcp.NewInput.func1%5Cn%5Ct/home/jason/go/src/github.com/elastic/beats/filebeat/input/tcp/input.go:59%5Cngithub.com/elastic/beats/filebeat/inputsource/tcp.(*client)).handle\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/inputsource/tcp/client.go:71\[ngithub.com/elastic/beats/filebeat/inputsource/tcp.(\*Server](http://ngithub.com/elastic/beats/filebeat/inputsource/tcp.(*Server)).run.func1\n\t/home/jason/go/src/github.com/elastic/beats/filebeat/inputsource/tcp/server.go:99"} filebeat stays running but does not accept the connections. filebeats.conf filebeat.inputs: - type: tcp max\_message\_size: 256MiB host: "localhost:514" processors: - drop\_fields: fields: ["host", "beat.hostname"] output.redis: hosts: ["redis1", "redis2"] key: "fastly" logging.level: info logging.to\_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 rotateeverybytes: 1073741824 ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [May 30, 2018, 8:11am UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/2 "2018-05-30T08:11:29Z")
Thanks for your report. I've opened an issue in beats repo > ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [May 30, 2018, 8:56am UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/3 "2018-05-30T08:56:41Z")
Hi @emrith After reviewing the code, the chance of an overflow comes to mind. Is it possible that during this time, filebeat has indexed 2^31 events? 🙂 ---
**Author:** ![emrith](https://avatars.discourse-cdn.com/v4/letter/e/76d3ee/32.png) [@emrith](https://discuss.elastic.co/u/emrith)\ **Post date:** [May 30, 2018, 2:59pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/4 "2018-05-30T14:59:47Z")
Yes we've defiantly are over 2^31. we have 3 filebeats instances that processed 10.8G events over the duration before they crashed. ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [May 30, 2018, 3:16pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/5 "2018-05-30T15:16:23Z")
Thanks for the confirmation. We've identified the problem and are working on a fix right now. ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [May 30, 2018, 6:02pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/6 "2018-05-30T18:02:22Z")
@emrith I've created a PR to fix that issue at [https://github.com/elastic/beats/pull/7214](https://github.com/elastic/beats/pull/7214), thanks for reporting this! May I ask you what type of log you are indexing over TCP? ---
**Author:** ![emrith](https://avatars.discourse-cdn.com/v4/letter/e/76d3ee/32.png) [@emrith](https://discuss.elastic.co/u/emrith)\ **Post date:** [May 31, 2018, 2:27pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/7 "2018-05-31T14:27:07Z")
syslog network request from an external CDN ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 28, 2018, 2:27pm UTC](https://discuss.elastic.co/t/filebeats-6-3-tcp-client-crash/133759/8 "2018-06-28T14:27:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.