How filebeat calculate metrics returned by HTTP endpoint

Elastic Stack Beats
1

Hi, it is unclear to me if the data is accumulated since the start of filebeat or it is a real-time view. Will the metrics be cleared if filebeat is restarted? Could you point to me the script that generates the metrics?

I'm trying to check if there is any failed or dropped events in filebeat by looking at output.events.dropped. If filebeat is restarted, will the number of dropped events be cleared out? I'm asking this because I want to run a monitoring script on filebeat and don't want it to keep failing on this condition.

By the way, is it possible to have dropped and failed events if the output is logstash? Doesn't filebeat guarantee at-least-once-delievery?

Thanks in advance!

    "output": {
      "events": {
        "acked": 0,
        "active": 0,
        "batches": 0,
        "dropped": 0,
        "duplicates": 0,
        "failed": 0,
        "total": 0
      },
      "read": {
        "bytes": 0,
        "errors": 0
      },
      "type": "logstash",
      "write": {
        "bytes": 0,
        "errors": 0
      }
    },
    "pipeline": {
      "clients": 1,
      "events": {
        "active": 944,
        "dropped": 0,
        "failed": 0,
        "filtered": 2,
        "published": 944,
        "retry": 0,
        "total": 946
      },
      "queue": {
        "acked": 0
      }
    }
  },
2

Yes, the number of dropped events are cleared out on restart. The counters are not persisted when Filebeat is stopped. So this means that metrics will be reset every time the Beat is restarted.

It seems that at the moment Logstash cannot report the number of dropped events.

Filebeat guarantees at-least-once delivery, for example if the output becomes unreachable, it keeps messages that was not forwarded in its memory or saves it to disk (depending on the configuration). Then FB attempts to send these events again. These are called failed events. Dropped events are event which were filtered out or events which cannot be indexed. These events are not sent again later.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.