Logstash configuration to delete input files once processed

Elastic Stack Logstash
1

I am trying to delete input files from directory once it was processed by Filebeat. Also wanted to confirm from filebeat that particular file is already processed so it is safe to delete. For that I tried few things on my conf file.
input {
file{
path => "C:/../../../*"
mode => "read"
sincedb_path => "NULL"
file_completed_action => "delete"
}
pipeline {
address => bamboolog
}
}

filter {
}

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
pipeline => abc
index => "indexName"
action => "update"
}
stdout { codec => json_lines }
}

Can anyone please suggest or help what I am doing wrong because I am getting below error -

[2022-12-27T13:02:07,730][ERROR][logstash.javapipeline ] Pipeline error {:pipeline_id=>"bamboolog", :exception=>#<LogStash::ConfigurationError: Specifying action => 'update' needs a document_id.>,
[2022-12-27T13:02:07,757][ERROR][logstash.agent ] Failed to execute action {:id=>:bamboolog, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2022-12-27T13:05:57,991][WARN ][org.logstash.plugins.pipeline.PipelineBus] Attempted to send event to 'bamboolog' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

2

This is your error, in your elasticsearch output you set the action to be update, to use in this way you need to provide the value for the _id field of the document using the document_id option.

You need to set document_id to use the field with the _id of the document.

3

Here is my full conf file. I added document_id in output. I got error and files were deleted from the input folder.Ideally it should delete files once filebeat would also up and file would processed.

input {
file{
path => "C:/../../../*"
mode => "read"
sincedb_path => "NULL"
file_completed_action => "delete"
}
pipeline {
address => bamboolog
}
}

filter {
mutate {
gsub => [ "message", "\t", " "]
}
dissect {
mapping => {
"message" => "%{level} %{timestampInLog} %{+timestampInLog} %{message_info}"
}
tag_on_failure => ["_dissectlevel_timestamp_bamboo"]
}
date {
match => [ "timestampInLog", "dd-MMM-yyyy HH:mm:ss","ISO8601" ]
target => "@timestamp"
tag_on_failure => [ "_dateparsefailure" ]
}
grok {
match => {
"[log][file][path]" => 'C:\..\..\..\%{GREEDYDATA:filename}.log'
}
tag_on_failure => ["_grokparsefailure"]
}
dissect {
mapping => {
"filename" => '%{Project}-%{Plan}-%{Job}-%{Build}'
}
tag_on_failure => ["_dissectlevel_filename"]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
pipeline => preprocess
index => "bamboolog-filebeat"
action => "update"
document_id => "%{[document][Build]}"
}
}

4

What error you got?

Also, use the preformatted button when sharing configurations, it makes it easier to understand, it is the </> button.

Sorry, just saw this now.

This is not possible, Logstash has no way to know if a File was already processed by filebeat or not, if you configure logstash to read a file that filebeat is also reading, and use the read mode, logstash will try to delete it when it finish processing.

5

Got some reference from this link - Deleting log files after they have finished processing
Firstly I tried to do this using Filebeat but then while searching found this above link and tried to do that using Logstash.
Please suggest what\how I could achieve that..?

6

There is nothing that I could suggest besides the already linked post.

Filebeat can not delete log files, only Logstash, if you need to delete log files after processing you need to use Logstash or use some external tool to delete the files after some time.

7

On Windows, if you do not want the in-memory sincedb persisted across restarts then set sincedb_path => "NUL". Setting it to NULL will persist it in a file called NULL in logstash's working directory.

8

@leandrojmp , I used Logstash to delete files and it does the same but before processing files from filebeat. But you mentioned - "Filebeat can not delete log files, only Logstash, if you need to delete log files after processing you need to use Logstash", but it is not doing as expected.

9

@Badger I used sincedb_path => "NULL" in my conf file. Above is my full conf file posted.
image

10

As I already said, Logstash has no way to know if Filebeat has processed the log or not, if you want to ingest a file and delete it, you need to use only Logstash, you can't use both as Logstash will keep deleting the files.

Check the previous answer.

11

Yes, I understand that. It is very unlikely that that is the right configuration.

12

Thank you @leandrojmp and @Badger for your help.. I was able to solve and achieve what I was looking for.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.