# Filebeat losing pointer and reading file from start

**URL:** <https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 26, 2018, 2:45am UTC](https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331 "2018-06-26T02:45:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)\
**Post date:** [June 26, 2018, 2:45am UTC](https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331/1 "2018-06-26T02:45:59Z")

</div>

Beats: 6.2.3

Have using filebeat for most of the our ELK solution to stream log files in to Elastic. IN one of the scenario, filebeat is reading the file from start.

scenario:  
On a secure server, have `rsync` service shipping the log files to shared drive. Every time `rsync` is executed, `delta` logs is copied over to shared drive. And then the permission of the file is changed along with group.

Filebeat is polling the shared drive log file, and every time `rsync` is appending to the log file, Filebeat losses its pointer and starts reading the log file from top; thus causing duplicate entries into elastic.

Rsync command is as below

```
find /opt/appa/logs/ -mindepth 1 -mtime +7 -delete
rsync -ar /opt/appa/logs/ /shareddrv/appa/logs/testhost/ -o
chgrp appgrp /shareddrv/appa/logs/testhost/*
chmod -R 0770 /shareddrv/appa/logs/testhost/*

```

Is this a `filebeat` issue or the way `rsync` is appending the file?  
Any body else seen this problem?

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [June 26, 2018, 2:50pm UTC](https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331/2 "2018-06-26T14:50:00Z")

</div>

Hello @paano, Filebeat internally uses the file inode to know if it's a new file or not.

I think by default the sync operation will by default creates new files and you have to use `--append` or `--append-verify` to append the data to the file instead of creating a new file.

---

<div class="post-metadata">

**Author:** ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)\
**Post date:** [June 26, 2018, 11:27pm UTC](https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331/3 "2018-06-26T23:27:58Z")

</div>

@pierhugues yes, i am using `-a` option to append the file.

`rsync -ar /opt/appa/logs/ /shareddrv/appa/logs/testhost/ -o`

Should be using the word `--append`

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [June 27, 2018, 1:25am UTC](https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331/4 "2018-06-27T01:25:38Z")

</div>

Yes, from my rsync man page

```auto
        -a, --archive archive mode; same as -rlptgoD (no -H)

```

I am pretty sure `--append` is different.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2018, 1:27am UTC](https://discuss.elastic.co/t/filebeat-losing-pointer-and-reading-file-from-start/137331/5 "2018-07-25T01:27:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.