Get pprof from filebeat,and "publish.events" is not equals to "libbeat.publisher.published_events"

[yingqin678]

i got somet metrics from filebeat by httpprof,and some of them are confusing me.
filebeat version:5.3.0
I got three times of metrics:
i want to know:what`s the meaning of publish.event? Why publish.event is smaller than libbeat.publisher.published_events in some condition while in other condition is bigger than libbeat.publisher.published_events?
{
"libbeat.output.kafka.bytes_write": 134,
"libbeat.output.kafka.bytes_read": 300,
"libbeat.kafka.call_count.PublishEvents": 199791,
"libbeat.kafka.published_and_acked_events": 0,
"libbeat.kafka.published_but_not_acked_events": 121672110,
"libbeat.outputs.messages_dropped": 0,
"libbeat.publisher.messages_in_worker_queues": 0,
"libbeat.publisher.published_events": 609,
"publish.events": 0,
}

{
"libbeat.kafka.call_count.PublishEvents": 82,
"libbeat.kafka.published_and_acked_events": 50380,
"libbeat.kafka.published_but_not_acked_events": 0,
"libbeat.outputs.messages_dropped": 0,
"libbeat.publisher.messages_in_worker_queues": 0,
"libbeat.publisher.published_events": 50380,
"publish.events": 165888,
}

{
"libbeat.kafka.call_count.PublishEvents": 2,
"libbeat.kafka.published_and_acked_events": 1225,
"libbeat.kafka.published_but_not_acked_events": 0,
"libbeat.outputs.messages_dropped": 0,
"libbeat.publisher.messages_in_worker_queues": 0,
"libbeat.publisher.published_events": 1225,
"publish.events": 2048,
}

[steffens (Steffen Siering)]

publish.events is # events forwarded from filebeat to libbeat. Have you enabled publish_async? checking the code, it seems publish.events is only used if publish_async: false (btw. it's not recommended to use publish_async: true)

libbeat.publisher.published_events is # of events after filtering

call_count.PublishEvents is # of batches of events being processed by kafka output

published_and_acked_events is # of events being ACKed by kafka

published_but_not_acked_events is # of events failed when sending to kafka.

Due to resends on failure the call count and not acked events can be much bigger then total number of events being published to libbeat.