Filebeat monitoring metrics sent bytes and throughput shown as 0

[daniel_fablius]

Hi, when i loadtest and monitoring filebeat, i found out that the throughput (/s) and sent bytest shown as 0, while the Event (/s) showing some value and also the logs data is shown in Kibana. which then i assume, the connection from filebeat to elasticsearch is not having any problem.

im wondering why the thorughput shown 0/s ?
is there any configuration needed for the throughput show any value? or maybe i misunderstand what the throughput supposed to represent?

im using filbeat and es 7.9.0

[Mario_Castro]

Hi @daniel_fablius

Can you provide a bit more information about the overall setup? It looks like a Filebeat bug. How are you monitoring Filebeat exactly? Can you post the configuration?

[daniel_fablius]

Hi @Mario_Castro

here is my yaml file for filebeat config

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config  
  namespace: logging
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    cluster.name: logging
    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          templates:
            - config:
                - type: container
                  paths: 
                    - /var/log/containers/*${data.kubernetes.container.id}.log

                  processors:
                    - decode_json_fields:
                        fields: ['message']
                        target: ""
                        add_error_key: true
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_URL}'
      index: "filebeat-%{+yyyy.MM.dd}"
    xpack.monitoring:
      enabled: true
    setup.ilm.enabled: false
    setup.template:
      name: "filebeat"   
      pattern: "filebeat-*"
      settings.index:
        number_of_shards: 9
        number_of_replicas: 1

i assume this lines below is the only one that matters for monitoring?

    xpack.monitoring:
      enabled: true

is there anything i missed here?

[daniel_fablius]

Hallo, anyone having the same issue with this?
is it actually bug? or im missing something? still cant find any configuration that might be contributing to this issue, as other metrics is showing except for filebeat sent bytes and throughtput that goes to ES (the one that goes to logstash have the metrics shown)

[shaunak (Shaunak Kashyap)]

This looks very much like this issue: https://github.com/elastic/beats/issues/20752. It's a bug in Beats that needs fixing.

Shaunak

[EugRomanchenko (Eugeny Romanchenko)]

Apparently this problem should have been solved in version 7.9.2 (look at https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-7.9.2.html). But after we have updated to this version, the problem is still present.