Delete file logstash

Elastic Stack Logstash
1

Hi,
I have a small problem i have logs and i read it with filebeat
filebeat.yml

- type: log
  enabled: true
  paths:
    - D:\elastic_stack\LOGS\CMS_LOGS\*
  fields:
        kafka_topic: "kafka-topic-cms-test1"
  multiline.type: pattern
  multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 15000

logstash.conf

output{
if ("Format log different" in [New log format]) {
      file {
        path => "D:/elastic_stack/LOGS/other/test-%{+YYYY-MM-dd}.txt"
        codec => line { format => "custom format: %{message}"}
      }
    }
}

So in the same time i want to delete the file that exist under D:\elastic_stack\LOGS\CMS_LOGS\ i mean how to delete file in the output block in logstash after creating another file like i did

Any help would be sincerely appreciate!
Thanks.

2

Hi,

According to this post: Deleting log files after they have finished processing - Meta Elastic / Elastic Tips and Common Fixes - Discuss the Elastic Stack.
It's not possible to delete a file in filebeat after processing it. You need to take the file in logsatsh input and use the same configuration as shown in the post I share.

Cad.

1 Like
3

Thanks for the reply Cad
indeed my output filebeat is kafka so my logstash input is kafka it's not possible to delete a file after filtring i mean in the output block ?
logstash.conf

input {
    kafka {
    bootstrap_servers => "127.0.0.1:9093,127.0.0.1:9094,127.0.0.1:9095"
    topics => ["kafka-topic-a-test1","kafka-topic-b-test1","kafka-topic-c-test1","kafka-topic-d-test1"]
    decorate_events => true
    group_id => "logstash-multi-topic-consumers"
    consumer_threads => 5
  }
}
output{
if [message] =~ /^\d{4}-\d{2}:\d{2}:\d{2}:\d{3}/ {
      file {
        path => "D:/elastic_stack/LOGS/other/test-%{+YYYY-MM-dd}.txt"
        codec => line { format => "custom format: %{message}"}
      }
    }
}

The file added successfully, i just want to delete the file when filebeat reads it exactly under D:\elastic_stack\LOGS\CMS_LOGS* I hope you understand me.
I added mode => "read" file_completed_action => "delete" it gives me an error Unknown setting 'mode' for file Unknown setting 'file_completed_action' for file

Thanks!

4

From what i know, no it is not possible.

In your case i think you can only use logstash and so remove the file read after processing it. Why are you using filebeat ?

Cad.

5

I have 5 dirs in each dir i have files in different format so i used filebeat to read all files in that 5 dir and i did the multiline to split them each log and send those logs to kafka then logstash to do the filtering in each log then i send it to Elasticsearch.
And delete file, I just want it if there is new log format in this directory, do it automatically in another directory and delete it from directory 0

Filebeat - collects logs and forwards them to a Kafka topic.

Kafka - brokers the data flow and queues it.

Logstash - aggregates the data from the Kafka topic, processes it and ships to Elasticsearch.

Elasticsearch - indexes the data.

Kibana - for analyzing the data.

is my architecture wrong? :confused:

6

Hi,

Sorry fo the late response.
Your architecture don't look wrong for me, i juste wanted to know if it was possible to you to only use ELK use the remove_file option of logstash.

I don't understand this line, you wnat to remove files under specific conditions ?

Cad.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.