### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 27, 2022, 12:14pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/2 "2022-12-27T12:14:29Z")
> [@BhawanaSharma](#):
>
> Pipeline error {:pipeline\_id=\>"bamboolog", :exception=\>#\ 'update' needs a document\_id.\>
This is your error, in your elasticsearch output you set the `action` to be `update`, to use in this way you need to provide the value for the `_id` field of the document using the `document_id` option.
You need to set `document_id` to use the field with the `_id` of the document.
---
### Author:  [@BhawanaSharma](https://discuss.elastic.co/u/BhawanaSharma)
#### Post date: [December 27, 2022, 3:19pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/3 "2022-12-27T15:19:47Z")
Here is my full conf file. I added document\_id in output. I got error and files were deleted from the input folder.Ideally it should delete files once filebeat would also up and file would processed.
input {
file{
path =\> "C:/../../../\*"
mode =\> "read"
sincedb\_path =\> "NULL"
file\_completed\_action =\> "delete"
}
pipeline {
address =\> bamboolog
}
}
filter {
mutate {
gsub =\> ["message", "\t", " "]
}
dissect {
mapping =\> {
"message" =\> "%{level} %{timestampInLog} %{+timestampInLog} %{message\_info}"
}
tag\_on\_failure =\> ["\_dissectlevel\_timestamp\_bamboo"]
}
date {
match =\> ["timestampInLog", "dd-MMM-yyyy HH:mm:ss","ISO8601"]
target =\> "@timestamp"
tag\_on\_failure =\> ["\_dateparsefailure"]
}
grok {
match =\> {
"[log][file][path]" =\> 'C:\..\..\..\%{GREEDYDATA:filename}.log'
}
tag\_on\_failure =\> ["\_grokparsefailure"]
}
dissect {
mapping =\> {
"filename" =\> '%{Project}-%{Plan}-%{Job}-%{Build}'
}
tag\_on\_failure =\> ["\_dissectlevel\_filename"]
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
manage\_template =\> false
pipeline =\> preprocess
index =\> "bamboolog-filebeat"
action =\> "update"
document\_id =\> "%{[document][Build]}"
}
}
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 27, 2022, 3:44pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/4 "2022-12-27T15:44:32Z")
What error you got?
Also, use the preformatted button when sharing configurations, it makes it easier to understand, it is the `` button.
> [@BhawanaSharma](#):
>
> I am trying to delete input files from directory once it was processed by Filebeat. Also wanted to confirm from filebeat that particular file is already processed so it is safe to delete.
Sorry, just saw this now.
This is not possible, Logstash has no way to know if a File was already processed by filebeat or not, if you configure logstash to read a file that filebeat is also reading, and use the `read` mode, logstash will try to delete it when it finish processing.
---
### Author:  [@BhawanaSharma](https://discuss.elastic.co/u/BhawanaSharma)
#### Post date: [December 27, 2022, 4:00pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/5 "2022-12-27T16:00:02Z")
Got some reference from this link - [Deleting log files after they have finished processing](https://discuss.elastic.co/t/deleting-log-files-after-they-have-finished-processing/246382)
Firstly I tried to do this using Filebeat but then while searching found this above link and tried to do that using Logstash.
Please suggest what\how I could achieve that..?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 27, 2022, 4:25pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/6 "2022-12-27T16:25:35Z")
There is nothing that I could suggest besides the already linked post.
Filebeat can not delete log files, only Logstash, if you need to delete log files after processing you need to use Logstash or use some external tool to delete the files after some time.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 27, 2022, 5:59pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/7 "2022-12-27T17:59:27Z")
> [@BhawanaSharma](#):
>
> sincedb\_path =\> "NULL"
On Windows, if you do not want the in-memory sincedb persisted across restarts then set `sincedb_path => "NUL"`. Setting it to NULL will persist it in a file called NULL in logstash's working directory.
---
### Author:  [@BhawanaSharma](https://discuss.elastic.co/u/BhawanaSharma)
#### Post date: [December 28, 2022, 5:59am UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/8 "2022-12-28T05:59:57Z")
@leandrojmp , I used Logstash to delete files and it does the same but before processing files from filebeat. But you mentioned - "Filebeat can not delete log files, only **Logstash, if you need to delete log files after processing you need to use Logstash**", but it is not doing as expected.
---
### Author:  [@BhawanaSharma](https://discuss.elastic.co/u/BhawanaSharma)
#### Post date: [December 28, 2022, 6:02am UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/9 "2022-12-28T06:02:21Z")
@Badger I used sincedb\_path =\> "NULL" in my conf file. Above is my full conf file posted.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 28, 2022, 1:11pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/10 "2022-12-28T13:11:56Z")
> [@BhawanaSharma](#):
>
> I used Logstash to delete files and it does the same but before processing files from filebeat
As I already said, Logstash has no way to know if Filebeat has processed the log or not, if you want to ingest a file and delete it, you need to use **only** Logstash, you can't use both as Logstash will keep deleting the files.
Check the previous answer.
> [@leandrojmp](#):
>
> This is not possible, Logstash has no way to know if a File was already processed by filebeat or not, if you configure logstash to read a file that filebeat is also reading, and use the `read` mode, logstash will try to delete it when it finish processing.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 28, 2022, 3:51pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/11 "2022-12-28T15:51:15Z")
> [@BhawanaSharma](#):
>
> I used sincedb\_path =\> "NULL" in my conf file
Yes, I understand that. It is very unlikely that that is the right configuration.
---
### Author:  [@BhawanaSharma](https://discuss.elastic.co/u/BhawanaSharma)
#### Post date: [January 11, 2023, 6:27am UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/12 "2023-01-11T06:27:43Z")
Thank you @leandrojmp and @Badger for your help.. I was able to solve and achieve what I was looking for.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 8, 2023, 6:27am UTC](https://discuss.elastic.co/t/logstash-configuration-to-delete-input-files-once-processed/322029/13 "2023-02-08T06:27:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.