Filebeat INFO output - libbeat.pipeline.events.filtered

Elastic Stack Beats
1

Hello
Watching filebeat run in the console with '-e' option, I can see it spitting out "INFO Non-zero metrics in the last 30s:"

I looked closer at some of the values:

beat.info.uptime.ms=29996
beat.memstats.gc_next=26967824
beat.memstats.memory_alloc=20770720
beat.memstats.memory_total=5068039560
filebeat.events.added=144
filebeat.events.done=144
filebeat.harvester.closed=3
filebeat.harvester.open_files=0
filebeat.harvester.running=0
filebeat.harvester.started=3
libbeat.config.module.running=0
libbeat.output.read.bytes=1404
libbeat.output.write.bytes=112515
libbeat.pipeline.clients=1
libbeat.pipeline.events.active=0
libbeat.pipeline.events.filtered=6
libbeat.pipeline.events.published=138
libbeat.pipeline.events.total=144
libbeat.pipeline.queue.acked=138
registrar.states.current=343
registrar.states.update=144
registrar.writes=6

I could not find much info on what these values mean, but I am particularly interested in the one called 'libbeat.pipeline.events.filtered' because it seems to suggest 6 of my log lines didn't make it through to be published. I could be wrong but looking for clarification... (and a way to find the problem with those 6 lines if indeed they are not being published)

Filebeat version 6.1, json log input and ES output

Thanks
Richard

2

Please share the complete Filebeat configuration you are using.

3

I believe this metric gets incremented with a processor intentional drops an event. Are you using processors?

1 Like
4

Hi, thanks for looking. Here's the config

filebeat:
  prospectors:
    -
      paths:
        - /data/logs/json/*.json
      input_type: log
      close_inactive: 5m
      close_eof: true
      clean_inactive: 25h
      ignore_older: 24h
      harvester_limit: 2
      scan.sort: "filename"
      json.keys_under_root: false

setup.template.name: "edge"
setup.template.pattern: "edge-*"
setup.template.fields: "template.json"

output.elasticsearch:
  hosts: ["http://x.x.x.x:9200"]
  index: "edge-nginx-%{+yyyy.MM.dd}"
  worker: 8
  bulk_max_size: 4096
5

No processors are used in the config so I'm surprised that libbeat.pipeline.events.filtered is > 0.

@steffens What would cause libbeat.pipeline.events.filtered > 0 ?

6

It may be caused by the empty lines. And there are many other events, like files state updated that u can find in Prospector.updateState.

7

Right, filebeat creates empty events on file state updates. These should not be published to the output, but must be processed (ACKed) in order with the other events, so to have the registry file updated in order. These empty events are filtered out by the publisher pipeline.

It's some 'internals' we hope to get rid of in the future.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.