Filebeat not sending Apache logs to Logstash

Elastic Stack Beats
1

Hello, All,

I have an issue where I have filebeat installed on my apache server with the following config:

filebeat:
  prospectors:
    - enabled: true
      backoff: 1s
      backoff_factor: 2
      clean_inactive: 0
      clean_removed: true
      close_eof: false
      close_inactive: 5m
      close_removed: true
      close_renamed: false
      close_timeout: 0
      document_type: apache
      encoding: plain
      fields_under_root: true
      harvester_buffer_size: 16384
      input_type: log
      max_backoff: 10s
      max_bytes: 10485760
      paths:
        - /var/log/httpd/*
      scan_frequency: 10s
      tags:
        - apache
      tail_files: false

My filebeat.yml is:

fields_under_root: false
filebeat.config_dir: /etc/filebeat/conf.d
filebeat.idle_timeout: 5s
filebeat.publish_async: false
filebeat.registry_file: .filebeat
filebeat.shutdown_timeout: 0
filebeat.spool_size: 2048
max_procs: ~
name: myserver
output.logstash:
  hosts:
    - "logstash_server:5044"
queue_size: 1000

So I installed filebeat way later than when I configured my apache server. Logrotation is set for /var/log/httpd/ where files are zipped everyday at 3am. Currently my filebeat can only send logs starting from today but is not sending logs older than today. I understand that filebeat can't harvest .gz files, so I unzipped those files and restarted filebeat and still cant see older apache logs in kibana.

In filebeat log, I see that it says that it has started harvesting the unzipped log files but nothing more after that.

Any help would be much appreciated. Thanks.

2

Hi! Are the old logs also under /var/log/httpd/* directory? Also do you have ignore_older parameter disabled?

3

hi @Kaiyan_Sheng,

Thank you for your response.

Yes the logs are also in the same directory path as /var/log/httpd/*. I have not set ignore_older parameter anywhere.

I thought if ignore_older is not set then by default it is set to disable ?

4

Hmmm have you ingested these old logs before? Maybe remove the registry file will help!

5

These old logs were never ingested. I have removed the registry file again and restarted filebeat, still now joy.