# Remote log file

**URL:** <https://discuss.elastic.co/t/remote-log-file/53725>\
**Category:** Beats\
**Created:** [June 23, 2016, 2:33am UTC](https://discuss.elastic.co/t/remote-log-file/53725 "2016-06-23T02:33:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![franzwong](https://avatars.discourse-cdn.com/v4/letter/f/f17d59/32.png) [@franzwong](https://discuss.elastic.co/u/franzwong)\
**Post date:** [June 23, 2016, 2:33am UTC](https://discuss.elastic.co/t/remote-log-file/53725/1 "2016-06-23T02:33:10Z")

</div>

I have a URL pointing to a log file. I can write a cron job to download (http get) this log file periodically. But I am just thinking whether filebeat will reset its cursor to the file after download. Also will it send duplicate line to ELK or do I need to remove duplicate on ELK side?

Or is there any other tool or way to do that? Thanks 🙂

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 23, 2016, 4:38am UTC](https://discuss.elastic.co/t/remote-log-file/53725/2 "2016-06-23T04:38:47Z")

</div>

If you can write a script to download the log file and only append the new lines you should be fine. Actually, if you just overwrite the existing file from the beginning to the end you should get that effect, but when I say overwrite I don't mean `curl > file.log` because that'll effectively truncate the file first and that'll get you into trouble. The existing file needs to be opened and the current log needs to be overwritten in place.

---

<div class="post-metadata">

**Author:** ![franzwong](https://avatars.discourse-cdn.com/v4/letter/f/f17d59/32.png) [@franzwong](https://discuss.elastic.co/u/franzwong)\
**Post date:** [June 23, 2016, 3:53pm UTC](https://discuss.elastic.co/t/remote-log-file/53725/3 "2016-06-23T15:53:17Z")

</div>

Thanks for reply. I will try that.

So it will detect duplicate and not to add that twice, right?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [June 24, 2016, 7:19am UTC](https://discuss.elastic.co/t/remote-log-file/53725/4 "2016-06-24T07:19:49Z")

</div>

Filebeat remembers the offset and in case you only append lines, no duplicates are sent. Filebeat has the at least once principle so in case of network problems it could still happen that duplicates are sent, but this is not related to reading the file.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2016, 2:33am UTC](https://discuss.elastic.co/t/remote-log-file/53725/5 "2016-07-14T02:33:16Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.