# Send the updated lines of file to logstash without re-running the filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 5, 2017, 12:37pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347 "2017-12-05T12:37:34Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![npasha](https://avatars.discourse-cdn.com/v4/letter/n/9dc877/32.png) [@npasha](https://discuss.elastic.co/u/npasha)\ **Post date:** [December 5, 2017, 12:37pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/1 "2017-12-05T12:37:34Z")
I configured the filebeat.yml with the prospector and sends the output to logstash parse and then send to elastic search. All worked fine. My question is if i append or modify the file, is it possible to send only the added lines to logstash without rerunning the filebeat? ---
**Author:** ![Mojster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mojster/32/21209_2.png) [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Post date:** [December 5, 2017, 12:41pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/2 "2017-12-05T12:41:14Z")
Filebeat marks the file end. So if new line are added it takes only this ones. Also if you stop/start filebeat it will continue. I assume the added lines should be at the bottom, otherwise it won't work. ---
**Author:** ![npasha](https://avatars.discourse-cdn.com/v4/letter/n/9dc877/32.png) [@npasha](https://discuss.elastic.co/u/npasha)\ **Post date:** [December 5, 2017, 1:06pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/3 "2017-12-05T13:06:12Z")
thanks for the reply, do i need to set "scan\_frequency" in filebeat.yml to look for the new data? ---
**Author:** ![npasha](https://avatars.discourse-cdn.com/v4/letter/n/9dc877/32.png) [@npasha](https://discuss.elastic.co/u/npasha)\ **Post date:** [December 5, 2017, 1:18pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/4 "2017-12-05T13:18:25Z")
-\>I tried by adding a new line but did not updated i.e. did not sent the new line to logstash. ``` I added at the bottom of the file. ``` -\>When we stop the filebeat, how can it continue to send the data. this is my filebeat.yml config file filebeat.prospectors: - type: log and logstashconfiguration file is input { beats { port =\> 5044 } } filter { } output { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } stdout { codec =\> rubydebug } } ---
**Author:** ![Mojster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mojster/32/21209_2.png) [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Post date:** [December 5, 2017, 1:45pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/5 "2017-12-05T13:45:47Z")
Filebeat overview in general: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) More explained: [https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html](https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html) By logic, filebeat must be running to continue shipping data. ---
**Author:** ![Mojster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mojster/32/21209_2.png) [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Post date:** [December 5, 2017, 1:58pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/6 "2017-12-05T13:58:07Z")
No you don't need to set `scan_frequency`. I'm always running on default. ---
**Author:** ![npasha](https://avatars.discourse-cdn.com/v4/letter/n/9dc877/32.png) [@npasha](https://discuss.elastic.co/u/npasha)\ **Post date:** [December 6, 2017, 7:10am UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/7 "2017-12-06T07:10:08Z")
Thanks a lot for the reply, filebeat is sending the data to logstash properly when i add some contents at the end, but for some files, i can see twice the data is collected in logstash, may i know why this happens? ![fl](https://us1.discourse-cdn.com/elastic/original/3X/2/3/23b86aca43e8350e5c49747ce072c90a84a63e75.PNG) and why the error "failed to publish event...." as highlighted is coming? ---
**Author:** ![Mojster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mojster/32/21209_2.png) [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Post date:** [December 6, 2017, 9:27am UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/8 "2017-12-06T09:27:27Z")
I have never had duplicates and I'm running on more than 900 000 000 log entries. So you must have duplicate files/entries or maybe did you clean the filebeatData so that he reloads all again. About the error. I don't mind it. Because all data still goes through. I think this appears on shutdown fo logstash/filebeat. Maybe someone else can help you further. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 6, 2017, 2:45pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/9 "2017-12-06T14:45:29Z")
Filebeat uses send-at-least-once semantics. That is, upon failure filebeat has to send again. This can lead to duplicates. There is no such thing as send-exactly-once. Send-exactly-once is normally emulated by by checkpointing are some kind of deduplication (e.g. sequence numbers as used in TCP). You can try to deduplicate by creating a message fingerprint or id (e.g. host+filename+timestamp+offset) and check if the id has been seen in the past. If you directly push to elasticsearch, setting the `_id` given on events contents will give you some deduplication in Elasticsearch. The [action type](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action) 'create' will fail if id is already known and [action type](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action) `index` will overwrite old document. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2018, 2:45pm UTC](https://discuss.elastic.co/t/send-the-updated-lines-of-file-to-logstash-without-re-running-the-filebeat/110347/10 "2018-01-03T14:45:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.