# Filebeat dropping events
**URL:** https://discuss.elastic.co/t/filebeat-dropping-events/271653
**Category:** Beats
**Tags:** filebeat
**Created:** [April 29, 2021, 1:39pm UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653 "2021-04-29T13:39:44Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@smanoranjan005](https://discuss.elastic.co/u/smanoranjan005)
#### Post date: [April 29, 2021, 1:39pm UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/1 "2021-04-29T13:39:44Z")
Hi,
I am using filebeat as daemonset in Kubernetes to transfer application logs to Logstash to Elasticsearch. But the issue here is there is some data drop during this transfer to ES.
There is only one filter in filebeat configmap to read only application namespace data.
I am validating the data on ES with the pod logs and found lots of data loss.
If there are 4 pods running for an application, I am not getting all 4 pods logs in ES.
I have checked filebeat & ES logs, but not able to find any error related to event rejection from ES.
I have tried to save filebeat output in the file and found the data loss there also.
I am using the latest(7.12) version of filebeat, logstash & elasticsearch.
Please help me here.
---
### Author:  [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 29, 2021, 5:08pm UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/2 "2021-04-29T17:08:41Z")
can you provide the configs for Filebeat and Logstash?
---
### Author:  [@smanoranjan005](https://discuss.elastic.co/u/smanoranjan005)
#### Post date: [April 30, 2021, 3:21am UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/3 "2021-04-30T03:21:09Z")
**Filebeat config:**
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/\*.log
processors:
- add\_kubernetes\_metadata:
host: ${NODE\_NAME}
matchers:
- logs\_path:
logs\_path: "/var/log/containers/"
- drop\_event:
when:
equals:
kubernetes.namespace: "istio-system"
- drop\_event:
when:
equals:
kubernetes.namespace: "kube-system"
- drop\_event:
when:
equals:
kubernetes.namespace: "kube-public"
- drop\_event:
when:
equals:
kubernetes.namespace: "monitoring"
setup.ilm.enabled: false
multiline.type: pattern
multiline.pattern: '^['
multiline.negate: false
multiline.match: after
output.elasticsearch:
hosts: ['logstash:5044']
**Logstash config:**
input {
beats {
port =\> "5044"
}
}
filter {
grok {
match =\> {"message" =\> ["[AUDIT] %{GREEDYDATA:message}"]}
overwrite =\> ["message"]
add\_tag =\> ["audit"]
}
}
output {
if "audit" in [tags] {
elasticsearch {
hosts =\> ["node1", "node2", "node3"]
index =\> "audit-%{+YYYY.MM.dd}"
}
}
if [kubernetes][container][name] == "app1" {
elasticsearch {
hosts =\> ["node1", "node2", "node3"]
index =\> "app1-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app2" {
elasticsearch {
hosts =\> ["node1", "node2", "node3"]
index =\> "app2-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] in ["app5", "app6", "app6-1", "app7", "app8", "app9", "app10", "app11"] {
elasticsearch {
hosts =\> ["node1", "node2", "node3"]
index =\> "inventory-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app4" {
elasticsearch {
hosts =\> ["node1", "node2", "node3"]
index =\> "app4-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts =\> ["node1", "node2", "node3"]
index =\> "filebeat-%{+YYYY.MM.dd}"
}
}
}
The count of concurrent files filebeat should read in our my environment will be 100-120.
---
### Author:  [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 30, 2021, 11:13am UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/4 "2021-04-30T11:13:24Z")
Can u please use the code format option so it's easier to read.
---
### Author:  [@smanoranjan005](https://discuss.elastic.co/u/smanoranjan005)
#### Post date: [April 30, 2021, 12:37pm UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/5 "2021-04-30T12:37:46Z")
Filebeat config:
```auto
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event:
when:
equals:
kubernetes.namespace: "istio-system"
- drop_event:
when:
equals:
kubernetes.namespace: "kube-system"
- drop_event:
when:
equals:
kubernetes.namespace: "kube-public"
- drop_event:
when:
equals:
kubernetes.namespace: "monitoring"
processors:
- drop_fields:
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "version", "kubernetes.labels.pod-template-hash", "kubernetes.pod.uid", "kubernetes.replicaset.name", "log.file.path", "log.offset", "kubernetes.node.name", "kubernetes.namespace", "kubernetes.labels.tier"]
setup.ilm.enabled: false
multiline.type: pattern
multiline.pattern: '^\['
multiline.negate: false
multiline.match: after
output.logstash:
hosts: ['logstash:5044']
```
Logstash Config:
```auto
input {
beats {
port => "5044"
}
}
filter {
grok {
match => {"message" => ["[AUDIT] %{GREEDYDATA:message}"]}
overwrite => ["message"]
add_tag => ["audit"]
}
}
output {
if "audit" in [tags] {
elasticsearch {
hosts => ["node1", "node2", "node3"]
index => "audit-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app1" {
elasticsearch {
hosts => ["node1", "node2", "node3"]
index => "app1-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app2" {
elasticsearch {
hosts => ["node1", "node2", "node3"]
index => "app2-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] in ["app5", "app6", "app6-1", "app7", "app8", "app9", "app10", "app11"] {
elasticsearch {
hosts => ["node1", "node2", "node3"]
index => "inventory-%{+YYYY.MM.dd}"
}
}
else if [kubernetes][container][name] == "app4" {
elasticsearch {
hosts => ["node1", "node2", "node3"]
index => "app4-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["node1", "node2", "node3"]
index => "filebeat-%{+YYYY.MM.dd}"
}
}
}
```
---
### Author:  [@smanoranjan005](https://discuss.elastic.co/u/smanoranjan005)
#### Post date: [May 6, 2021, 8:04am UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/6 "2021-05-06T08:04:00Z")
Any luck here?
---
### Author:  [@smanoranjan005](https://discuss.elastic.co/u/smanoranjan005)
#### Post date: [May 17, 2021, 7:50am UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/7 "2021-05-17T07:50:38Z")
Any update?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 14, 2021, 9:50am UTC](https://discuss.elastic.co/t/filebeat-dropping-events/271653/8 "2021-06-14T09:50:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.