# Problem with Filebeat and Kafka

**URL:** <https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 10, 2019, 9:14am UTC](https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988 "2019-10-10T09:14:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![saisimo02](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@saisimo02](https://discuss.elastic.co/u/saisimo02)\
**Post date:** [October 10, 2019, 9:14am UTC](https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988/1 "2019-10-10T09:14:05Z")

</div>

Hello,

I have a problem with my configuration filebeat+kafka+logstash.

I have a lot of files every minute and I am using kafka between filebeat and logstash to deal with that.

Filebeat send an error message:

```
2019-10-10T10:35:03.708+0200 ERROR file/states.go:112 State for /home/test/Documents/PRODUCERS/Data/PmProducer/demo-C2019-10-10T10:20:00+02:00-2019-10-10T10:21:00+02:00_calv-vm25.lker.fr.xml should have been dropped, but couldn't as state is not finished.
2019-10-10T10:35:03.708+0200 ERROR file/states.go:112 State for /home/test/Documents/PRODUCERS/Data/PmProducer/demo-C2019-10-10T10:21:00+02:00-2019-10-10T10:22:00+02:00_calv-vm26.lker.fr.xml should have been dropped, but couldn't as state is not finished.
2019-10-10T10:35:03.708+0200 ERROR file/states.go:112 State for /home/test/Documents/PRODUCERS/Data/PmProducer/demo-C2019-10-10T10:21:00+02:00-2019-10-10T10:22:00+02:00_calv-vm41.lker.fr.xml should have been dropped, but couldn't as state is not finished.
2019-10-10T10:35:03.708+0200 ERROR file/states.go:112 State for /home/test/Documents/PRODUCERS/Data/PmProducer/demo-C2019-10-10T10:20:00+02:00-2019-10-10T10:21:00+02:00_calv-vm30.lker.fr.xml should have been dropped, but couldn't as state is not finished.

```

my filebeat input:

```
- type: log
  enabled: true
  paths:
- /home/test/Documents/PRODUCERS/Data/PmProducer/*robustness*.xml
  fields:
  # level: debug
     name: hdfs-pm-robu
  document_type: hdfs-pm-robu
  multiline.pattern: '<measInfo'
  multiline.negate: true
  ignore_older: 2m
  scan_frequency: 15s
  multiline.match: after
  close_inactive: 3m
  clean_inactive: 7m
  clean_removed: true

```

My filebeat output:

```
output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["kafka1:9092","kafka2:9092","kafka3:9092"]

  # message topic selection + partitioning
  topic: '%{[fields.name]}'
  partition.round_robin:
reachable_only: false

  #required_acks: 1
  #compression: gzip
  max_message_bytes: 100000000

```

When I have a data every minute I am getting this error message and there is data loss, when I generate data every 3 minutes there is no more data loss even if there is the same size of data in the two cases.

 ![filebeat](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fcfd0cd28250cddaa97a4fb876dc583a19231b8b.png)

Thank you for help

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [October 10, 2019, 6:27pm UTC](https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988/2 "2019-10-10T18:27:32Z")

</div>

> [@saisimo02](#):
>
> 3.708+0200 ERR

You should set clean\_inactive to a higher value otherwise it can cause data loss.  
That may be the causing the problem.

 ![clean_inactive](https://us1.discourse-cdn.com/elastic/original/3X/5/4/5470d910d1dae1a330023fd1723eec4374d8df22.png)

---

<div class="post-metadata">

**Author:** ![saisimo02](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@saisimo02](https://discuss.elastic.co/u/saisimo02)\
**Post date:** [October 11, 2019, 8:23am UTC](https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988/3 "2019-10-11T08:23:35Z")

</div>

Thank you for your reply, but I am still having the same problem. I set clean\_inactive to 10m and commented other parameters.

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [October 11, 2019, 9:02pm UTC](https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988/4 "2019-10-11T21:02:54Z")

</div>

What other params did you comment out ?  
Can you please post your latest filebeat.xml ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2019, 9:03pm UTC](https://discuss.elastic.co/t/problem-with-filebeat-and-kafka/202988/5 "2019-11-08T21:03:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.