Filebeat - On-Demand Loading Compressed (.gz) Files from Stdin

If you want to load compressed files like .gz .zip etc. you can use Filebeat with input stdin.

vasek ~ $ filebeat version
filebeat version 7.0.1 (amd64), libbeat 7.0.1 [cbffb4dcc8d1d2b0ef2078cb7d7546092ee86e57 built 2019-04-29 12:09:21 +0000 UTC]

This command creates file /etc/filebeat/filebeat-stdin.yml.

cat << "EOF" > /etc/filebeat/filebeat-stdin.yml
- type: stdin
  fields_under_root: true
    redis-key: "stdin-nazev-aplikace" "nazev-aplikace"
    application.environment: "production"
    service.type: "application" "tomcat"
logging.level: warning
logging.to_files: true
logging.to_syslog: false
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 1
  permissions: 0640
  rotateeverybytes: 10485760
  metrics.enabled: false
 - drop_fields:
     fields: [ "fields", "which", "you", "want", "to", "drop" ]
  events: 4096
  flush.timeout: 10 2048
  hosts: ["localhost:6379"]
  key: "%{[redis-key]:unknown}"
  • Filebeat reads data from STDIN
  • Some fields will be added to document
  • Some fields can be removed from document
  • Data will be sent to Redis queue = output

There is command for loading compress files to filebeat stdin:

 zcat oracle-logs.2019-05-*.log.gz | filebeat -e -c /etc/filebeat/filebeat-stdin.yml

I hope it will be useful for someone.

Hello, thanks for reaching out and providing a solution for loading compressed files. I'm certain this will help someone in the future. Feel free to let us know if you have questions.


1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.