# Delete file logstash **URL:** \ **Category:** Logstash\ **Created:** [April 13, 2022, 10:09am UTC](https://discuss.elastic.co/t/delete-file-logstash/302314 "2022-04-13T10:09:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Post date:** [April 13, 2022, 10:09am UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/1 "2022-04-13T10:09:52Z")
Hi, I have a small problem i have logs and i read it with filebeat filebeat.yml ```auto - type: log enabled: true paths: - D:\elastic_stack\LOGS\CMS_LOGS\* fields: kafka_topic: "kafka-topic-cms-test1" multiline.type: pattern multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}- multiline.negate: true multiline.match: after multiline.max_lines: 15000 ``` logstash.conf ```auto output{ if ("Format log different" in [New log format]) { file { path => "D:/elastic_stack/LOGS/other/test-%{+YYYY-MM-dd}.txt" codec => line { format => "custom format: %{message}"} } } } ``` So in the same time i want to delete the file that exist under D:\elastic\_stack\LOGS\CMS\_LOGS\ i mean how to delete file in the output block in logstash after creating another file like i did Any help would be sincerely appreciate! Thanks. ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [April 13, 2022, 1:20pm UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/2 "2022-04-13T13:20:24Z")
Hi, According to this post: [Deleting log files after they have finished processing - Meta Elastic / Elastic Tips and Common Fixes - Discuss the Elastic Stack](https://discuss.elastic.co/t/deleting-log-files-after-they-have-finished-processing/246382). It's not possible to delete a file in filebeat after processing it. You need to take the file in logsatsh input and use the same configuration as shown in the post I share. Cad. ---
**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Post date:** [April 13, 2022, 1:39pm UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/3 "2022-04-13T13:39:11Z")
Thanks for the reply Cad indeed my output filebeat is kafka so my logstash input is kafka it's not possible to delete a file after filtring i mean in the output block ? logstash.conf ```auto input { kafka { bootstrap_servers => "127.0.0.1:9093,127.0.0.1:9094,127.0.0.1:9095" topics => ["kafka-topic-a-test1","kafka-topic-b-test1","kafka-topic-c-test1","kafka-topic-d-test1"] decorate_events => true group_id => "logstash-multi-topic-consumers" consumer_threads => 5 } } output{ if [message] =~ /^\d{4}-\d{2}:\d{2}:\d{2}:\d{3}/ { file { path => "D:/elastic_stack/LOGS/other/test-%{+YYYY-MM-dd}.txt" codec => line { format => "custom format: %{message}"} } } } ``` The file added successfully, i just want to delete the file when filebeat reads it exactly under **D:\elastic\_stack\LOGS\CMS\_LOGS\*** I hope you understand me. I added `mode => "read" file_completed_action => "delete"` it gives me an error **Unknown setting 'mode' for file Unknown setting 'file\_completed\_action' for file** Thanks! ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [April 13, 2022, 2:31pm UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/4 "2022-04-13T14:31:52Z")
> [@alex\_vermex](#): > > it's not possible to delete a file after filtring i mean in the output block ? From what i know, no it is not possible. In your case i think you can only use logstash and so remove the file read after processing it. Why are you using filebeat ? Cad. ---
**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Post date:** [April 13, 2022, 7:41pm UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/5 "2022-04-13T19:41:28Z")
I have 5 dirs in each dir i have files in different format so i used filebeat to read all files in that 5 dir and i did the multiline to split them each log and send those logs to kafka then logstash to do the filtering in each log then i send it to Elasticsearch. And delete file, I just want it if there is new log format in this directory, do it automatically in another directory and delete it from directory 0 Filebeat - collects logs and forwards them to a Kafka topic. Kafka - brokers the data flow and queues it. Logstash - aggregates the data from the Kafka topic, processes it and ships to Elasticsearch. Elasticsearch - indexes the data. Kibana - for analyzing the data. is my architecture wrong? 😕 ---
**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\ **Post date:** [April 21, 2022, 8:47am UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/6 "2022-04-21T08:47:57Z")
Hi, Sorry fo the late response. Your architecture don't look wrong for me, i juste wanted to know if it was possible to you to only use ELK use the remove\_file option of logstash. > [@alex\_vermex](#): > > And delete file, I just want it if there is new log format in this directory, do it automatically in another directory and delete it from directory 0 I don't understand this line, you wnat to remove files under specific conditions ? Cad. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 19, 2022, 8:48am UTC](https://discuss.elastic.co/t/delete-file-logstash/302314/7 "2022-05-19T08:48:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.