Filebeat Stats and Metrics

Elastic Stack Beats
, ,
1

Hi folks, I am trying to better understand the metrics I am currently seeing from the filebeat /stats call.

Specifically looking to understand exactly what these metrics are:

    u'filebeat': {u'events': {u'active': 12, u'added': 571, u'done': 559},
    u'libbeat':
                u'output': {u'events': {u'acked': 516,
                                       u'active': 0,
                                        u'batches': 54,
                                        u'dropped': 0,
                                        u'duplicates': 0,
                                        u'failed': 0,
                                        u'total': 516},
                u'read': {u'bytes': 324, u'errors': 0},
                              u'type': u'logstash',
                              u'write': {u'bytes': 98123, u'errors': 0}},
                 u'pipeline': {u'clients': 1,
                              u'events': {u'active': 12,
                                                 u'dropped': 0,
                                                 u'failed': 0,
                                                 u'filtered': 43,
                                                 u'published': 528,
                                                 u'retry': 288,
                                                 u'total': 571},
                              u'queue': {u'acked': 516}}}

I trimmed down to just the sections I really care about.

  • What do the "dropped" metrics mean?
  • What do the "output" and the "pipeline" mean?
  • What exactly is an "event"?
  • What is the difference between a filebeat "event" and a libbeat "event"? The two sets of numbers are different and I'm not quite sure why

In addition to those questions, if I could get definitions for the different stats that would be great!

2

Hello @Emily_Hontoria

I am not a Beat developer so the provided information might not be accurate.

I think this post might explain the dropped meaning: Safely publish an event using libbeat - #3 by steffens

An example of an event dropped in libbeat output is a document which is malformed (e.g. bad encoding) when being sent to a final destination (e.g. Elasticsearch).
An example of an event dropped in libbeat pipeline is a document which is excluded (e.g. exclude_lines in log input or drop_event processor.

We document those fields:

  • In GoDoc
  • The exported fields when monitoring Filebeat using Metricbeat Filebeat module (doc)

I think:

  • the output stats are related to the declared output in the configuration (e.g. Beats are able to send to different destinations)
  • the pipeline stats are related to the declared processors, filters, conditionals and the internal queue

We document those fields in GoDoc.

I think the difference is the following:

  • a filebeat event is an event with specific fields, respecting the filebeat definition of event (e.g. a filebeat event coming from syslog tcp input with its dedicated fields or from a log input), built on top of a libbeat event
  • a libbeat event is the common event format shared by all beats. Every event must have a timestamp and provide encodable Fields in Fields

In any case, those metrics are meant to be digested by the Elastic Stack Monitoring tools (see here).

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.