Filebeat missing some log lines

Elastic Stack Beats
1

Hi Everyone ,

I had been in situation where I m sending logs to logstash shipper using filebeat ,

It seems filebeat is skipping some events , especially event at around 23:59 to 00:01 . I am not sure anyone else facing same issue ,

My filebeat config file is

filebeat:
  prospectors:
    -
      paths:
        - /path_of_log_file
      input_type: log
      fields_under_root: true
      fields:
        type: php_log
        server_group: webgroup
output:
  logstash:
    hosts: ["X.X.X.X:5046","X.X.X.X:5047"]
    loadbalance: true

logging:
  to_syslog: false
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  selectors: ["*" ]
  level: error

Thanks

2

Are you using log rotation around midnight? What kind of log ration are you using? What happens to the rotated files?

3

It was strange , there was no log rotation at those servers where log was missed .

also my architecture is something like this -

But for general information what will be impact of log rotation we generally use logrotate feature of linux and how to overcome from these kind of failures

filebeat -> shipper -> kafka <- indexer -> elastic <- kibana

4

If you use the general log rotation there shouldn't be an issue. Filebeat finishes reading the old file and picks up the new one.

I assume shipper and index above are both LS instances. Any chance to share the log files of filebeat around the time the events went missing? Does it happen every day or it happened only once? Which version of the following are you using?

  • Filebeat
  • Logstash
  • beat-input-plugin