Filebeat Stats and Metrics

Hi folks, I am trying to better understand the metrics I am currently seeing from the filebeat /stats call.

Specifically looking to understand exactly what these metrics are:

    u'filebeat': {u'events': {u'active': 12, u'added': 571, u'done': 559},
                u'output': {u'events': {u'acked': 516,
                                       u'active': 0,
                                        u'batches': 54,
                                        u'dropped': 0,
                                        u'duplicates': 0,
                                        u'failed': 0,
                                        u'total': 516},
                u'read': {u'bytes': 324, u'errors': 0},
                              u'type': u'logstash',
                              u'write': {u'bytes': 98123, u'errors': 0}},
                 u'pipeline': {u'clients': 1,
                              u'events': {u'active': 12,
                                                 u'dropped': 0,
                                                 u'failed': 0,
                                                 u'filtered': 43,
                                                 u'published': 528,
                                                 u'retry': 288,
                                                 u'total': 571},
                              u'queue': {u'acked': 516}}}

I trimmed down to just the sections I really care about.

  • What do the "dropped" metrics mean?
  • What do the "output" and the "pipeline" mean?
  • What exactly is an "event"?
  • What is the difference between a filebeat "event" and a libbeat "event"? The two sets of numbers are different and I'm not quite sure why

In addition to those questions, if I could get definitions for the different stats that would be great!

Hello @Emily_Hontoria

I am not a Beat developer so the provided information might not be accurate.

I think this post might explain the dropped meaning: Safely publish an event using libbeat

An example of an event dropped in libbeat output is a document which is malformed (e.g. bad encoding) when being sent to a final destination (e.g. Elasticsearch).
An example of an event dropped in libbeat pipeline is a document which is excluded (e.g. exclude_lines in log input or drop_event processor.

We document those fields:

  • In GoDoc
  • The exported fields when monitoring Filebeat using Metricbeat Filebeat module (doc)

I think:

  • the output stats are related to the declared output in the configuration (e.g. Beats are able to send to different destinations)
  • the pipeline stats are related to the declared processors, filters, conditionals and the internal queue

We document those fields in GoDoc.

I think the difference is the following:

  • a filebeat event is an event with specific fields, respecting the filebeat definition of event (e.g. a filebeat event coming from syslog tcp input with its dedicated fields or from a log input), built on top of a libbeat event
  • a libbeat event is the common event format shared by all beats. Every event must have a timestamp and provide encodable Fields in Fields

In any case, those metrics are meant to be digested by the Elastic Stack Monitoring tools (see here).

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.