Filebeat - On-Demand Loading Compressed (.gz) Files from Stdin

Elastic Stack Beats
1

If you want to load compressed files like .gz .zip etc. you can use Filebeat with input stdin.

vasek ~ $ filebeat version
filebeat version 7.0.1 (amd64), libbeat 7.0.1 [cbffb4dcc8d1d2b0ef2078cb7d7546092ee86e57 built 2019-04-29 12:09:21 +0000 UTC]

This command creates file /etc/filebeat/filebeat-stdin.yml.

cat << "EOF" > /etc/filebeat/filebeat-stdin.yml
filebeat.inputs:
- type: stdin
  fields_under_root: true
  fields:
    redis-key: "stdin-nazev-aplikace"
    application.name: "nazev-aplikace"
    application.environment: "production"
    service.type: "application"
    service.name: "tomcat"
logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 1
  permissions: 0640
  rotateeverybytes: 10485760
  metrics.enabled: false
processors:
 - drop_fields:
     fields: [ "fields", "which", "you", "want", "to", "drop" ]
queue.mem:
  events: 4096
  flush.timeout: 10
  flush.min.events: 2048
output.redis:
  hosts: ["localhost:6379"]
  key: "%{[redis-key]:unknown}"
EOF
  • Filebeat reads data from STDIN
  • Some fields will be added to document
  • Some fields can be removed from document
  • Data will be sent to Redis queue = output

There is command for loading compress files to filebeat stdin:

 zcat oracle-logs.2019-05-*.log.gz | filebeat -e -c /etc/filebeat/filebeat-stdin.yml

I hope it will be useful for someone.

4 Likes
2

Hello, thanks for reaching out and providing a solution for loading compressed files. I'm certain this will help someone in the future. Feel free to let us know if you have questions.

Thanks.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.