How to configure the output section in logstash conf make copy of logs on my local disk

Elastic Stack Logstash
1

Hi,

How can i get copy or output of parsed logs in logstash on my local disk?

Our Case:
We are thinking not just send logs to Elasticsearch by logstash also to have output from logs on our disk, then we have backup/archive system of our logs.
In case of data loss/ELk failure we dont want to end up with losing some of the important events in our system.

So how we can implement this request?

2

Just add a file output to your output section. You'll probably want to set the codec to json_lines.

Why not just back up the Elasticsearch indexes instead of storing the log copies as flat files?

3

So this can be the solution

output {
file {
path => ...
codec => line { format => "custom format: %{message}"}
}
}

or Do you prefer second method Elasticsearch indexes' backup?

another question please. If i have log files for two years and i want the logstash just send the last two weeks logs to the elasticsearch, how we can do this?

4

So this can be the solution

If you want to capture the whole event the line codec won't do. My suggestion is still json_lines.

I'd prefer backing up the ES data.

If i have log files for two years and i want the logstash just send the last two weeks logs to the elasticsearch, how we can do this?

You can use a conditional wrapping a drop filter that compares the @timestamp field to a fixed timestamp.

5

Really i appreciate your efforts to help and make it easier for community

6

Hi @magnusbaeck, I tested your suggestion with json_lines and now i have file output on my disk by using following output code:
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
file {
path => "/path/backup-logs/BackupSyslog-%{+YYYY-MM-dd}.log"
#codec => line { format => "custom format: %{message}"}
#codec => "rubydebug"
codec => "json_lines"
}
}

I have two more questions:
1- How to compress this file output by using gzip / Gzip the output stream before writing to disk?
Is it correct to use this
gzip => true

where to use the drop filter? in output section or in filter section?
Please if you have sample script of using drop filter with elasticsearch output!

7

I have two more questions:
1- How to compress this file output by using gzip / Gzip the output stream before writing to disk?
Is it correct to use this
gzip => true

Yes.

where to use the drop filter? in output section or in filter section?

Filters are always in the filter section.

Please if you have sample script of using drop filter with elasticsearch output!

Have a look at the age filter: age | Logstash Reference [5.4] | Elastic

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.