# Filebeat stops processing files after 1 or 2 days issue

**URL:** <https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 31, 2016, 5:20am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402 "2016-10-31T05:20:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![elasticheart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elasticheart/32/65189_2.png) [@elasticheart](https://discuss.elastic.co/u/elasticheart)\
**Post date:** [October 31, 2016, 5:20am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402/1 "2016-10-31T05:20:12Z")

</div>

Hi,

I have Filebeat configured to fetch data from log files in a folder. I have log files, which is sliced to 10 mb log fies in my linux box, and so I configured filebeat to scan the entire folder. There will be approx 10k entries per day. I am forwarding filebeat entries to kafka, and from kafka to logstash, and finally to ES.

FB --\> KFK --\> LS --\> ES

but the thing is that, sometimes after 1 day after i start logstash, the process will be running, but there will be no indexes in elasticsearch for that day. Or sometimes, there will be 5 or 6 entries for that day. Once i terminate the filebeat process and restarts it, then the entry gets populated as normal and everything works fine.

Below is my filebeat configuration.

```
filebeat.prospectors:
- input_type: log
  paths:
    - /logfolder/logfile*
  exclude_files: [".lck"]
  fields:
    logtype: LOGFILE
  document_type: dummylog
  scan_frequency: 1m
  close_inactive: 10s
  multiline.pattern: '^<[A-Za-z_]{3} [[:digit:]]{2}, [[:digit:]]{4} ([[:digit:]]{1}|[[:digit:]]{2}):[[:digit:]]{2}:[[:digit:]]{2}:([[:digit:]]{1}|[[:digit:]]{2}|[[:digit:]]{3}) [A-Z]{2}>'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 5000
filebeat.registry_file: .regfile
output.kafka:
  enabled: true
  hosts: ["192.168.1.1:9092"]
  topic: filebeatlogs
  worker: 1
  max_retries: 2
output.file:
  enabled: true
  path: "/outputfolder/logdata"
  filename: filebeatlogs

```

How can I fix this?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [October 31, 2016, 7:48am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402/2 "2016-10-31T07:48:29Z")

</div>

Which version of Filebeat are you using? Which version of Logstash?

---

<div class="post-metadata">

**Author:** ![elasticheart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elasticheart/32/65189_2.png) [@elasticheart](https://discuss.elastic.co/u/elasticheart)\
**Post date:** [October 31, 2016, 8:15am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402/3 "2016-10-31T08:15:31Z")

</div>

filebeat-5.0.0-alpha5-linux-x86\_64  
logstash-2.3.4

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [October 31, 2016, 8:23am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402/4 "2016-10-31T08:23:17Z")

</div>

- Could you update to the 5.0 GA release?
- Can you share some log files with at least INFO level from filebeat?

---

<div class="post-metadata">

**Author:** ![elasticheart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elasticheart/32/65189_2.png) [@elasticheart](https://discuss.elastic.co/u/elasticheart)\
**Post date:** [October 31, 2016, 9:05am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402/5 "2016-10-31T09:05:33Z")

</div>

lemme update to 5.0 GA and reply. Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2016, 5:20am UTC](https://discuss.elastic.co/t/filebeat-stops-processing-files-after-1-or-2-days-issue/64402/6 "2016-11-21T05:20:24Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.