Zip file pass through

This may be a strange request, but is there a way that I can use Filebeat / Logstash to do a straight pass through of a zip file?

I have a site that has 10's of gigs of log files that I want to process a day, but they are on a slow connection. So I would like to zip the all log files together, then at the end of the day, I wanted Filebeat to take that zip file and send it up to my Logstash server to save it as a file.
Once I have the zip file on the Logstash server, I would probably use cron or something to unzip the files, and then do a local Filebeat send on the Logstash server to process the files.
Due to latency issues, I cannot stream the files all day long so that I am only doing a differential on the log files.

I know there are other ways to get zip files onto a server, but since I already have a pipeline setup for logs, I thought that I would try and use Filebeat and Logstash to handle that piece as well.

I attempted to create a pipeline to handle this:
Filebeat.yml

- type: log
  enabled: true
  paths:
    - /root/logs.zip
output.logstash:
  hosts: ["10.1.2.143:5044"]

Logstash conf:

input {
  pipeline {
    address => zip_file
  }
}

output {
        file {
                path => "/root/zip_test.zip"
        }
}

When I run through this setup, it takes a 1.3 MB zip file, then it processes the zip file and spits out an 8.3 MB zip file. It's actually kind of cool, it json'ifies my zip file, it adds a bunch of tags, and uses the binary from the zip as the message.

Does anyone have any suggestions?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.