Hi!
i'm running Filebeat 7.5.1 at the moment and now we are implementing the Centralized Management. It's going good (17 servers so far converted), but my latest one is giving me a headache...
I enrolled without a problem, apply the configuration tags and nothing happens. I get the green config status in the centralized management list for the server, but no logs are coming in.
When I SSH into the server and restart filebeat (sudo systemctl restart filebeat), I can see all the missing logs (since last restart) are being send over and appear in Kibana for the correct index.
BUT...... no new message are coming in???? So I turned on the debug mode for filebeat. But can somebody help me understand this debug message from Filebeat:
2020-02-07T11:00:11.728+0100 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://s008aa57:5000))
2020-02-07T11:00:11.729+0100 INFO pipeline/output.go:105 Connection to backoff(async(tcp://s008aa57:5000)) established
2020-02-07T11:00:13.813+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":8}},"total":{"ticks":1710,"time":{"ms":87},"value":1710},"user":{"ticks":1550,"time":{"ms":79}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"88b5308a-013a-4a08-bd70-f246c8305500","uptime":{"ms":540039}},"memstats":{"gc_next":20430752,"memory_alloc":12758640,"memory_total":133676344},"runtime":{"goroutines":48}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":4096,"total":4096},"read":{"errors":1},"write":{"bytes":242415}},"pipeline":{"clients":3,"events":{"active":4118,"retry":6144}}},"registrar":{"states":{"current":109}},"system":{"load":{"1":0.14,"15":0.06,"5":0.08,"norm":{"1":0.07,"15":0.03,"5":0.04}}}}}}
2020-02-07T11:00:41.766+0100 ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 10.1.35.71:48790->10.1.35.46:5000: i/o timeout
2020-02-07T11:00:41.819+0100 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected
2020-02-07T11:00:43.520+0100 ERROR pipeline/output.go:121 Failed to publish events: client is not connected
2020-02-07T11:00:43.521+0100 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://s008aa57:5000))
2020-02-07T11:00:43.522+0100 INFO pipeline/output.go:105 Connection to backoff(async(tcp://s008aa57:5000)) established
2020-02-07T11:00:43.812+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":{"ms":4}},"total":{"ticks":1810,"time":{"ms":98},"value":1810},"user":{"ticks":1640,"time":{"ms":94}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"88b5308a-013a-4a08-bd70-f246c8305500","uptime":{"ms":570038}},"memstats":{"gc_next":20430752,"memory_alloc":17539456,"memory_total":138457160},"runtime":{"goroutines":48}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":4096,"total":4096},"read":{"errors":1},"write":{"bytes":242882}},"pipeline":{"clients":3,"events":{"active":4118,"retry":6144}}},"registrar":{"states":{"current":109}},"system":{"load":{"1":0.09,"15":0.05,"5":0.07,"norm":{"1":0.045,"15":0.025,"5":0.035}}}}}}
I can see it says error, but nothing more!
What is wrong then?
If I go back to the non-centralized configuration and unenroll the server, it all works fine again!!??
Cheers
Tim