What's the exact meaning of log entries generated by filebeat 5.2.2

[kzhang (Kelvin)]

Hi, I am new to filebeat and doing some troubleshooting, I want to understand the meaning of log entrie in the filebeat.log generated by the filebeat 5.2.2.
Take the following log entries as an example, can anybody explain what these keys/values mean? like libbeat.logstash.call_count.PublishEvents=1,libbeat.logstash.published_and_acked_events=1, libbeat.publisher.published_events=1. Thank you in advance.

INFO Non-zero metrics in the last 30s: registrar.writes=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=3303 registrar.states.update=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.publisher.published_events=1 publish.events=1

[ruflin]

In general these are internal metrics which are collected by each beat. They are looked every 30s (default) on the info the level. The part that is logged is the diff from the last time it was logged. Assuming updates is a counter and it was 4 and 30s later it is 6, the output will be 2. Having said that, we are aware that this is only good for counters and has issues with total values which also decrease etc. We plan to improve this and also document it better, but didn't get to it yet.

For the values itself, some details on the above:

• libbeat.logstash.call_count.PublishEvents: How many times the internal PublishEvents function was called. That is mainly interesting for debugging.
• libbeat.logstash.published_and_acked_events: How many events were send AND acknowledge by LS. If some events fail, you will also see a non_acked counter.
• libbeat.publisher.published_events: How many events were published across all outputs during the last 30s.
• registrar.states.update: This is coming from filebeat (we should prefix it ...) and states how many times the internal states of the registry were updated

Some of the namings above will change in 6.0 and some will even be removed as only 1 output at the time can be enabled which simplifies thing. As you see in the above example, quite a few values are identical because only 1 output is enabled (I assume).