If you want to load compressed files like .gz .zip etc. you can use Filebeat with input stdin.
vasek ~ $ filebeat version
filebeat version 7.0.1 (amd64), libbeat 7.0.1 [cbffb4dcc8d1d2b0ef2078cb7d7546092ee86e57 built 2019-04-29 12:09:21 +0000 UTC]
This command creates file /etc/filebeat/filebeat-stdin.yml.
cat << "EOF" > /etc/filebeat/filebeat-stdin.yml
filebeat.inputs:
- type: stdin
fields_under_root: true
fields:
redis-key: "stdin-nazev-aplikace"
application.name: "nazev-aplikace"
application.environment: "production"
service.type: "application"
service.name: "tomcat"
logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 1
permissions: 0640
rotateeverybytes: 10485760
metrics.enabled: false
processors:
- drop_fields:
fields: [ "fields", "which", "you", "want", "to", "drop" ]
queue.mem:
events: 4096
flush.timeout: 10
flush.min.events: 2048
output.redis:
hosts: ["localhost:6379"]
key: "%{[redis-key]:unknown}"
EOF
- Filebeat reads data from STDIN
- Some fields will be added to document
- Some fields can be removed from document
- Data will be sent to Redis queue = output
There is command for loading compress files to filebeat stdin:
zcat oracle-logs.2019-05-*.log.gz | filebeat -e -c /etc/filebeat/filebeat-stdin.yml
I hope it will be useful for someone.