# Filebeat stop writing files(via file output) when the output file gets removed **URL:** \ **Category:** Beats\ **Created:** [April 12, 2017, 8:19am UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114 "2017-04-12T08:19:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sunwl](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@sunwl](https://discuss.elastic.co/u/sunwl)\ **Post date:** [April 12, 2017, 8:19am UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/1 "2017-04-12T08:19:21Z")
The scenario is that I need to read files, make some changes and write the content to another file (say output\_file) via filebeat. Also I had another application(say app2) to do the process-and-remove thing to the output\_file. I know it is strange but I need it. The problem is that when the filebeat is writing the output\_file and the output\_file is removed by app2 while writing, filebeat stops to write any more. For example. The original\_file is 100MB. FIlebeat reads the original\_file and writes it to output\_file. When the output\_file grows to 50MB, app2 removes it. Then the remaining 50MB gets lost. I tried with below script to generate output\_file and it works well. All 10000000 records are processed. **for ((i=0; i\<10000000; ++i)); do echo $i \>\> output\_file.log; done** Please give me a hand on this. Thanks. Best regards, Ryan ---
**Author:** ![maddin2016](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddin2016/32/16599_2.png) [@maddin2016](https://discuss.elastic.co/u/maddin2016)\ **Post date:** [April 12, 2017, 9:35am UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/2 "2017-04-12T09:35:36Z")
Try this option ```auto output.file: ... rotate_every_kb: 51200 number_of_files: 2 ``` With this option you don't need your second script. You will always have one file that is not bigger then 50 MB. ---
**Author:** ![sunwl](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@sunwl](https://discuss.elastic.co/u/sunwl)\ **Post date:** [April 12, 2017, 12:43pm UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/3 "2017-04-12T12:43:27Z")
Thanks for the answer. Splitting data into 2 files is not the purpose. I just want to solve it in my scenario. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 12, 2017, 2:56pm UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/4 "2017-04-12T14:56:50Z")
The scenario as described is not really clear to me. > read files, make some changes and write the content to another file - which process is reading the file. - what kind of changes are we talking about > process-and-remove thing process-and-remove thing? So the process aforementioned ETL process is filtering out lines or is it deleting files? > The problem is that when the filebeat is writing the output\_file and the output\_file is removed by app2 while writing, filebeat stops to write any more. So `app2` does some post-processing on filebeat output? Or is `app2` running instead of filebeat? Why does `app2` remove the file, filebeat is still writing to? Under which condition do you remove a file? Normally a file's metadata are only removed, but as long as a file is still held by any process, it's not really deleted yet. That is, you can not just delete a file and expect a process to create a new file, as the OS will not report any kind of error/signal to the process. The option given by `maddin2016` incorporates file-rotation. The idea is to rely on file-rotation to give you an idea when filebeat has finished writing to a log-file. Alternatively to relying on rotation in filebeat, you can configure filebeat to send events to stdout. Then you can process/pipe the output right from the stream, the way you want. ---
**Author:** ![sunwl](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@sunwl](https://discuss.elastic.co/u/sunwl)\ **Post date:** [April 13, 2017, 3:56am UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/5 "2017-04-13T03:56:32Z")
Sorry for my poor ability of expression. It seems that I make it complicated. The scenario is that: 1. Filebeat reads file\_input and writes data into file\_output 2 App2 processes file\_output and removes file\_output. Problem: When App2 removes file\_output and Filebeat is still writing this file, the remaining data gets lost. Thank you. Best regards, Ryan ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 13, 2017, 9:38am UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/6 "2017-04-13T09:38:41Z")
well, as I already wrote, you can't just delete a file a process is still writing to. The file does still exist under the hood, it's just that you can not find it anymore in the directory structure. That is, you have to wait with deleting the file until filebeat has finished writing to it... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 3, 2017, 8:19am UTC](https://discuss.elastic.co/t/filebeat-stop-writing-files-via-file-output-when-the-output-file-gets-removed/82114/7 "2017-05-03T08:19:25Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.