# Filebeat in endless loop

**URL:** <https://discuss.elastic.co/t/filebeat-in-endless-loop/142445>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 31, 2018, 9:34pm UTC](https://discuss.elastic.co/t/filebeat-in-endless-loop/142445 "2018-07-31T21:34:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ray\_Kim](https://avatars.discourse-cdn.com/v4/letter/r/54ee81/32.png) [@Ray\_Kim](https://discuss.elastic.co/u/Ray_Kim)\
**Post date:** [July 31, 2018, 9:34pm UTC](https://discuss.elastic.co/t/filebeat-in-endless-loop/142445/1 "2018-07-31T21:34:18Z")

</div>

I have filebeat reading s3 logs but it sometimes get stuck in an endless loop and sometimes it just hangs and (rarely) goes as expected.

```
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /home/linux/logs/s3-dev-logs/*
    - /home/linux/logs/cloudfront-dev-logs/*.gz
 close_timeout: 5m
 output.logstash:
   hosts: ["localhost:5044"]

```

What I mean by endless loop is that Filebeat just keeps re-reading the log files

All (Filebeat, Logstash and Elasticsearch) are 6.3.2.

The command that I run filebeat is:  
`./filebeat -e -once -c filebeat.yml -d "publish"`

and the os is rhls7.4

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [August 1, 2018, 4:59pm UTC](https://discuss.elastic.co/t/filebeat-in-endless-loop/142445/2 "2018-08-01T16:59:48Z")

</div>

> [@Ray\_Kim](#):
>
> /home/linux/logs/cloudfront-dev-logs/\*.gz

Are you trying to read gzip logs? AFAIK Filebeat does not handle this. You would need to extract them first. Probably you should be setting `exclude_files` to skip .gz.

> **[Log input | Filebeat Reference \[6.3\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-input-log.html#filebeat-input-log-exclude-files)**

---

<div class="post-metadata">

**Author:** ![Ray\_Kim](https://avatars.discourse-cdn.com/v4/letter/r/54ee81/32.png) [@Ray\_Kim](https://discuss.elastic.co/u/Ray_Kim)\
**Post date:** [August 1, 2018, 7:09pm UTC](https://discuss.elastic.co/t/filebeat-in-endless-loop/142445/3 "2018-08-01T19:09:19Z")

</div>

Even when I comment the .gz part out, it gets stuck in an endless loop most of the time. As an aside, the s3-dev-logs are ASCII text, if that makes any difference..  
Also, how can I kill filebeat, as control+c doesn't work and it just stays alive and I have to close the terminal session to kill it.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [August 2, 2018, 1:10am UTC](https://discuss.elastic.co/t/filebeat-in-endless-loop/142445/4 "2018-08-02T01:10:19Z")

</div>

If you use `-once` then you should also set `close_eof: true` in order for the harvester to exit. Once all havesters exit then the Beat should stop. See the description of the `-once` flag in the [docs](https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html).

In order for the harvester to stop the data also needs to be successfully delivered to the output.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2018, 1:10am UTC](https://discuss.elastic.co/t/filebeat-in-endless-loop/142445/5 "2018-08-30T01:10:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.