# FileBeat misses lines at the beginning of a new file

**URL:** <https://discuss.elastic.co/t/filebeat-misses-lines-at-the-beginning-of-a-new-file/82420>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 14, 2017, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-misses-lines-at-the-beginning-of-a-new-file/82420 "2017-04-14T14:46:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BertrandC4](https://avatars.discourse-cdn.com/v4/letter/b/9fc348/32.png) [@BertrandC4](https://discuss.elastic.co/u/BertrandC4)\
**Post date:** [April 14, 2017, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-misses-lines-at-the-beginning-of-a-new-file/82420/1 "2017-04-14T14:46:46Z")

</div>

Sometimes Filebeat (5.3) misses some lines at the beginning of a new file.  
From debug filebeat logs, it looks like filebeat detects an **Update existing file for harvesting** but it is **not** an upated/renamed/moved file, just a new file.

I imagine the os (centos 6.8) create a new log file with an inode equals to an already harvested and removed log file. Is that possible ?

FileBeat.yml :

```
filebeat.prospectors:
- paths:
  - /logm/*/logs/*.log
  
  
output.logstash:
  enabled: true
  hosts: ["server-1:5044"]

logging:
  level: debug
  to_files: true
  to_syslog: false
  files:
    path: logs
    name: a=filebeat~c=logm~h=server-1.log
    keepfiles: 100
    rotateeverybytes: 104857600

```

filebeat.log :

> 2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=5~l=69905.log  
> 2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=57~l=69905.log  
> 2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=57~l=69905.log, offset: 1182750  
> 2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=57~l=69905.log  
> 2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log  
> 2017-04-14T13:42:27Z DBG **Update existing file for harvesting** : /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log, **offset: 20479**  
> 2017-04-14T13:42:27Z DBG Resuming harvesting of file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log, offset: 20479  
> 2017-04-14T13:42:27Z DBG Set previous offset for file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log. Offset: 20479  
> 2017-04-14T13:42:27Z DBG Setting offset for file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log. Offset: 20479  
> 2017-04-14T13:42:27Z DBG Publish: {  
> "@timestamp": "2017-04-14T13:42:26.681Z",  
> "beat": {  
> "hostname": "server-1.novalocal",  
> "name": "server-1.novalocal",  
> "version": "5.3.0"  
> },  
> "input\_type": "log",  
> "message": "2017-04-14T13:41:09.434Z#00000000000000013328#00000000000001999200#QD.69ZyS9.nVd - - [03/Sep/2759:13:07:48 -0800] "GET /vskDmIv9H HTTP/1.0" 2124 6779",  
> "offset": 1999200,  
> "source": "/logm/test/logs/c=logm~a=test~h=server-1~f=52~l=69905.log",  
> "type": "log"  
> }  
> 2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/elasticsearch-e1/logs/logm-e1.log  
> 2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/elasticsearch-e1/logs/logm-e1.log, offset: 8768  
> 2017-04-14T13:42:27Z DBG File didn't change: /logm/elasticsearch-e1/logs/logm-e1.log  
> 2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=1~l=69905.log  
> 2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=1~l=69905.log, offset: 10485750  
> 2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=1~l=69905.log  
> 2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=28~l=69905.log  
> 2017-04-14T13:42:27Z INFO **Harvester started for file** : /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log  
> 2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=28~l=69905.log, offset: 5935050  
> 2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=28~l=69905.log

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [April 19, 2017, 8:29am UTC](https://discuss.elastic.co/t/filebeat-misses-lines-at-the-beginning-of-a-new-file/82420/2 "2017-04-19T08:29:46Z")

</div>

It seems like the same inode which was used for a previous file which was deleted, is now used again. Normally this can be solved by using `clean_removed: true`. But this is enabled by default.

What is the log rotating algorithm you are using?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 17, 2017, 8:41am UTC](https://discuss.elastic.co/t/filebeat-misses-lines-at-the-beginning-of-a-new-file/82420/3 "2017-05-17T08:41:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.