# File beat stop harvesting logs from few files suddenly , works fine after restart

**URL:** <https://discuss.elastic.co/t/file-beat-stop-harvesting-logs-from-few-files-suddenly-works-fine-after-restart/178286>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 24, 2019, 3:08pm UTC](https://discuss.elastic.co/t/file-beat-stop-harvesting-logs-from-few-files-suddenly-works-fine-after-restart/178286 "2019-04-24T15:08:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![praveen.gunupati](https://avatars.discourse-cdn.com/v4/letter/p/47e85d/32.png) [@praveen.gunupati](https://discuss.elastic.co/u/praveen.gunupati)\
**Post date:** [April 24, 2019, 3:08pm UTC](https://discuss.elastic.co/t/file-beat-stop-harvesting-logs-from-few-files-suddenly-works-fine-after-restart/178286/1 "2019-04-24T15:08:10Z")

</div>

We are using file beat version 5.6 to tail logs from all the docker containers of a machine and sending them to logstash.

Here is the config we have for file beat.

'''  
filebeat.prospectors:

- input\_type: log  
paths:
  - /var/lib/docker/containers/_/_.log\*  
fields:  
env: dev  
ignore\_older: 2h  
document\_type: docker  
scan\_frequency: 10s  
close\_inactive: 2h  
close\_removed: true  
json.keys\_under\_root: true  
json.overwrite\_keys: true  
json.add\_error\_key: true  
exclude\_files: ['.gz$']

output.logstash:  
hosts: ["logstash:5001"]  
timeout: 30s

max\_bytes: 104857600  
max\_message\_bytes: 104857600  
'''

Earlier we were using swarm as a orchestration tool with that each node was having 40-50 containers, the above configuration was working fine , we didn't see any issues with that.

After we moved to kubernetes the number of containers on the node is doubled. With that the file beat is stop harvesting or sending logs to logstash from few containers. We restart the filebeat on the specific node to send the logs to logstash again.After some time we are seeing the same issue again.

filebeat registry timestamp says it is tailed all the logs.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 25, 2019, 10:28am UTC](https://discuss.elastic.co/t/file-beat-stop-harvesting-logs-from-few-files-suddenly-works-fine-after-restart/178286/2 "2019-04-25T10:28:02Z")

</div>

Do you have any filebeat logs/metrics you can share?

Please format logs and configurations using the `</>` button or 3 backticks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2019, 10:28am UTC](https://discuss.elastic.co/t/file-beat-stop-harvesting-logs-from-few-files-suddenly-works-fine-after-restart/178286/3 "2019-05-23T10:28:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.