Filebeat dropping events

Elastic Stack Beats
1

Hi,

I am using filebeat as daemonset in Kubernetes to transfer application logs to Logstash to Elasticsearch. But the issue here is there is some data drop during this transfer to ES.
There is only one filter in filebeat configmap to read only application namespace data.
I am validating the data on ES with the pod logs and found lots of data loss.
If there are 4 pods running for an application, I am not getting all 4 pods logs in ES.

I have checked filebeat & ES logs, but not able to find any error related to event rejection from ES.
I have tried to save filebeat output in the file and found the data loss there also.

I am using the latest(7.12) version of filebeat, logstash & elasticsearch.

Please help me here.

2

can you provide the configs for Filebeat and Logstash?

3

Filebeat config:
filebeat.yml: |-
filebeat.inputs:

  • type: container
    paths:
    • /var/log/containers/*.log
      processors:
    • add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      • logs_path:
        logs_path: "/var/log/containers/"
    • drop_event:
      when:
      equals:
      kubernetes.namespace: "istio-system"
    • drop_event:
      when:
      equals:
      kubernetes.namespace: "kube-system"
    • drop_event:
      when:
      equals:
      kubernetes.namespace: "kube-public"
    • drop_event:
      when:
      equals:
      kubernetes.namespace: "monitoring"

setup.ilm.enabled: false
multiline.type: pattern
multiline.pattern: '^['
multiline.negate: false
multiline.match: after

output.elasticsearch:
hosts: ['logstash:5044']

Logstash config:
input {
beats {
port => "5044"
}
}
filter {
grok {
match => {"message" => ["[AUDIT] %{GREEDYDATA:message}"]}
overwrite => [ "message" ]
add_tag => ["audit"]
}
}
output {
if "audit" in [tags] {
elasticsearch {
hosts => [ "node1", "node2", "node3" ]
index => "audit-%{+YYYY.MM.dd}"
}
}
if [kubernetes][container][name] == "app1" {
elasticsearch {
hosts => [ "node1", "node2", "node3" ]
index => "app1-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app2" {
elasticsearch {
hosts => [ "node1", "node2", "node3" ]
index => "app2-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] in ["app5", "app6", "app6-1", "app7", "app8", "app9", "app10", "app11"] {
elasticsearch {
hosts => [ "node1", "node2", "node3" ]
index => "inventory-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app4" {
elasticsearch {
hosts => [ "node1", "node2", "node3" ]
index => "app4-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => [ "node1", "node2", "node3" ]
index => "filebeat-%{+YYYY.MM.dd}"
}
}
}

The count of concurrent files filebeat should read in our my environment will be 100-120.

4

Can u please use the code format option so it's easier to read.

5

Filebeat config:

filebeat.yml: |-
  filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"
      - drop_event:
          when:
            equals:
              kubernetes.namespace: "istio-system"
      - drop_event:
          when:
            equals:
              kubernetes.namespace: "kube-system"
      - drop_event:
          when:
            equals:
              kubernetes.namespace: "kube-public"
      - drop_event:
          when:
            equals:
              kubernetes.namespace: "monitoring"
  processors:
    - drop_fields:
          fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "version", "kubernetes.labels.pod-template-hash", "kubernetes.pod.uid", "kubernetes.replicaset.name", "log.file.path", "log.offset", "kubernetes.node.name", "kubernetes.namespace", "kubernetes.labels.tier"]

  setup.ilm.enabled: false
  multiline.type: pattern
  multiline.pattern: '^\['
  multiline.negate: false
  multiline.match: after

  output.logstash:
    hosts: ['logstash:5044']

Logstash Config:

input {
  beats {
    port => "5044"
  }
}
filter {
  grok {
    match => {"message" => ["[AUDIT] %{GREEDYDATA:message}"]}
    overwrite => [ "message" ]
    add_tag => ["audit"]
  }
}

output {
  if "audit" in [tags] {
    elasticsearch {
      hosts => [ "node1", "node2", "node3" ]
      index => "audit-%{+YYYY.MM.dd}"
    }
  }
  else if [kubernetes][container][name] == "app1" {
    elasticsearch {
      hosts => [ "node1", "node2", "node3" ]
      index => "app1-%{+YYYY.MM.dd}"
    }
  }
  else if [kubernetes][container][name] == "app2" {
    elasticsearch {
      hosts => [ "node1", "node2", "node3" ]
      index => "app2-%{+YYYY.MM.dd}"
    }
  }
  else if [kubernetes][container][name] in ["app5", "app6", "app6-1", "app7", "app8", "app9", "app10", "app11"] {
    elasticsearch {
      hosts => [ "node1", "node2", "node3" ]
      index => "inventory-%{+YYYY.MM.dd}"
    }
  }
  else if [kubernetes][container][name] == "app4" {
    elasticsearch {
      hosts => [ "node1", "node2", "node3" ]
      index => "app4-%{+YYYY.MM.dd}"
    }
  }
  else {
    elasticsearch {
      hosts => [ "node1", "node2", "node3" ]
      index => "filebeat-%{+YYYY.MM.dd}"
    }
  }
}
6

Any luck here?

7

Any update?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.