Filebeat debug message

Hi!

i'm running Filebeat 7.5.1 at the moment and now we are implementing the Centralized Management. It's going good (17 servers so far converted), but my latest one is giving me a headache...

I enrolled without a problem, apply the configuration tags and nothing happens. I get the green config status in the centralized management list for the server, but no logs are coming in.
When I SSH into the server and restart filebeat (sudo systemctl restart filebeat), I can see all the missing logs (since last restart) are being send over and appear in Kibana for the correct index.

BUT...... no new message are coming in???? So I turned on the debug mode for filebeat. But can somebody help me understand this debug message from Filebeat:

2020-02-07T11:00:11.728+0100	INFO	pipeline/output.go:95	Connecting to backoff(async(tcp://s008aa57:5000))
2020-02-07T11:00:11.729+0100	INFO	pipeline/output.go:105	Connection to backoff(async(tcp://s008aa57:5000)) established
2020-02-07T11:00:13.813+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":8}},"total":{"ticks":1710,"time":{"ms":87},"value":1710},"user":{"ticks":1550,"time":{"ms":79}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"88b5308a-013a-4a08-bd70-f246c8305500","uptime":{"ms":540039}},"memstats":{"gc_next":20430752,"memory_alloc":12758640,"memory_total":133676344},"runtime":{"goroutines":48}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":4096,"total":4096},"read":{"errors":1},"write":{"bytes":242415}},"pipeline":{"clients":3,"events":{"active":4118,"retry":6144}}},"registrar":{"states":{"current":109}},"system":{"load":{"1":0.14,"15":0.06,"5":0.08,"norm":{"1":0.07,"15":0.03,"5":0.04}}}}}}
2020-02-07T11:00:41.766+0100	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 10.1.35.71:48790->10.1.35.46:5000: i/o timeout
2020-02-07T11:00:41.819+0100	ERROR	logstash/async.go:256	Failed to publish events caused by: client is not connected
2020-02-07T11:00:43.520+0100	ERROR	pipeline/output.go:121	Failed to publish events: client is not connected
2020-02-07T11:00:43.521+0100	INFO	pipeline/output.go:95	Connecting to backoff(async(tcp://s008aa57:5000))
2020-02-07T11:00:43.522+0100	INFO	pipeline/output.go:105	Connection to backoff(async(tcp://s008aa57:5000)) established
2020-02-07T11:00:43.812+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":{"ms":4}},"total":{"ticks":1810,"time":{"ms":98},"value":1810},"user":{"ticks":1640,"time":{"ms":94}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"88b5308a-013a-4a08-bd70-f246c8305500","uptime":{"ms":570038}},"memstats":{"gc_next":20430752,"memory_alloc":17539456,"memory_total":138457160},"runtime":{"goroutines":48}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":4096,"total":4096},"read":{"errors":1},"write":{"bytes":242882}},"pipeline":{"clients":3,"events":{"active":4118,"retry":6144}}},"registrar":{"states":{"current":109}},"system":{"load":{"1":0.09,"15":0.05,"5":0.07,"norm":{"1":0.045,"15":0.025,"5":0.035}}}}}}

I can see it says error, but nothing more!
What is wrong then?

If I go back to the non-centralized configuration and unenroll the server, it all works fine again!!??

Cheers
Tim

Hi @TimTim - Before we begin, take note that Beats Central Management is discontinued. Please consider this before proceeding with your development / implementation.

Regarding your problem, it is a bit hard to tell from the logs you provided. I also do not see any DEBUG logs. Could you do run Filebeat with debug logging and provide the full log? For example:

  • filebeat -e -d "*"

Thank you.

OK, I missed that it is discontinued. Then I guess we just unenroll it and leave it as it was when it worked. Thanks.

PS: True, no DEBUG lines there, forgot to turn it on.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.