This may be a strange request, but is there a way that I can use Filebeat / Logstash to do a straight pass through of a zip file?
I have a site that has 10's of gigs of log files that I want to process a day, but they are on a slow connection. So I would like to zip the all log files together, then at the end of the day, I wanted Filebeat to take that zip file and send it up to my Logstash server to save it as a file.
Once I have the zip file on the Logstash server, I would probably use cron or something to unzip the files, and then do a local Filebeat send on the Logstash server to process the files.
Due to latency issues, I cannot stream the files all day long so that I am only doing a differential on the log files.
I know there are other ways to get zip files onto a server, but since I already have a pipeline setup for logs, I thought that I would try and use Filebeat and Logstash to handle that piece as well.
I attempted to create a pipeline to handle this:
Filebeat.yml
- type: log
enabled: true
paths:
- /root/logs.zip
output.logstash:
hosts: ["10.1.2.143:5044"]
Logstash conf:
input {
pipeline {
address => zip_file
}
}
output {
file {
path => "/root/zip_test.zip"
}
}
When I run through this setup, it takes a 1.3 MB zip file, then it processes the zip file and spits out an 8.3 MB zip file. It's actually kind of cool, it json'ifies my zip file, it adds a bunch of tags, and uses the binary from the zip as the message.
Does anyone have any suggestions?