I have an issue where I have filebeat installed on my apache server with the following config:
filebeat: prospectors: - enabled: true backoff: 1s backoff_factor: 2 clean_inactive: 0 clean_removed: true close_eof: false close_inactive: 5m close_removed: true close_renamed: false close_timeout: 0 document_type: apache encoding: plain fields_under_root: true harvester_buffer_size: 16384 input_type: log max_backoff: 10s max_bytes: 10485760 paths: - /var/log/httpd/* scan_frequency: 10s tags: - apache tail_files: false
My filebeat.yml is:
fields_under_root: false filebeat.config_dir: /etc/filebeat/conf.d filebeat.idle_timeout: 5s filebeat.publish_async: false filebeat.registry_file: .filebeat filebeat.shutdown_timeout: 0 filebeat.spool_size: 2048 max_procs: ~ name: myserver output.logstash: hosts: - "logstash_server:5044" queue_size: 1000
So I installed filebeat way later than when I configured my apache server. Logrotation is set for /var/log/httpd/ where files are zipped everyday at 3am. Currently my filebeat can only send logs starting from today but is not sending logs older than today. I understand that filebeat can't harvest .gz files, so I unzipped those files and restarted filebeat and still cant see older apache logs in kibana.
In filebeat log, I see that it says that it has started harvesting the unzipped log files but nothing more after that.
Any help would be much appreciated. Thanks.