Filebeat Stats and Metrics

[Emily_Hontoria]

Hi folks, I am trying to better understand the metrics I am currently seeing from the filebeat /stats call.

Specifically looking to understand exactly what these metrics are:

    u'filebeat': {u'events': {u'active': 12, u'added': 571, u'done': 559},
    u'libbeat':
                u'output': {u'events': {u'acked': 516,
                                       u'active': 0,
                                        u'batches': 54,
                                        u'dropped': 0,
                                        u'duplicates': 0,
                                        u'failed': 0,
                                        u'total': 516},
                u'read': {u'bytes': 324, u'errors': 0},
                              u'type': u'logstash',
                              u'write': {u'bytes': 98123, u'errors': 0}},
                 u'pipeline': {u'clients': 1,
                              u'events': {u'active': 12,
                                                 u'dropped': 0,
                                                 u'failed': 0,
                                                 u'filtered': 43,
                                                 u'published': 528,
                                                 u'retry': 288,
                                                 u'total': 571},
                              u'queue': {u'acked': 516}}}

I trimmed down to just the sections I really care about.

• What do the "dropped" metrics mean?
• What do the "output" and the "pipeline" mean?
• What exactly is an "event"?
• What is the difference between a filebeat "event" and a libbeat "event"? The two sets of numbers are different and I'm not quite sure why

In addition to those questions, if I could get definitions for the different stats that would be great!

[Luca_Belluccini]

Hello @Emily_Hontoria

I am not a Beat developer so the provided information might not be accurate.

@Emily_Hontoria

What do the "dropped" metrics mean?

I think this post might explain the dropped meaning: Safely publish an event using libbeat - #3 by steffens

An example of an event dropped in libbeat output is a document which is malformed (e.g. bad encoding) when being sent to a final destination (e.g. Elasticsearch).
An example of an event dropped in libbeat pipeline is a document which is excluded (e.g. exclude_lines in log input or drop_event processor.

We document those fields:

• In GoDoc
• The exported fields when monitoring Filebeat using Metricbeat Filebeat module (doc)

@Emily_Hontoria

What do the "output" and the "pipeline" mean?

I think:

• the output stats are related to the declared output in the configuration (e.g. Beats are able to send to different destinations)
• the pipeline stats are related to the declared processors, filters, conditionals and the internal queue

We document those fields in GoDoc.

@Emily_Hontoria

What is the difference between a filebeat "event" and a libbeat "event"?

I think the difference is the following:

• a filebeat event is an event with specific fields, respecting the filebeat definition of event (e.g. a filebeat event coming from syslog tcp input with its dedicated fields or from a log input), built on top of a libbeat event
• a libbeat event is the common event format shared by all beats. Every event must have a timestamp and provide encodable Fields in Fields

---

In any case, those metrics are meant to be digested by the Elastic Stack Monitoring tools (see here).