# Filebeat missing some log lines

**URL:** <https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 11, 2016, 1:28pm UTC](https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487 "2016-02-11T13:28:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\
**Post date:** [February 11, 2016, 1:28pm UTC](https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487/1 "2016-02-11T13:28:17Z")

</div>

Hi Everyone ,

I had been in situation where I m sending logs to logstash shipper using filebeat ,

It seems filebeat is skipping some events , especially event at around 23:59 to 00:01 . I am not sure anyone else facing same issue ,

My filebeat config file is

```auto
filebeat:
  prospectors:
    -
      paths:
        - /path_of_log_file
      input_type: log
      fields_under_root: true
      fields:
        type: php_log
        server_group: webgroup
output:
  logstash:
    hosts: ["X.X.X.X:5046","X.X.X.X:5047"]
    loadbalance: true

logging:
  to_syslog: false
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  selectors: ["*"]
  level: error

```

Thanks

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 11, 2016, 1:49pm UTC](https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487/2 "2016-02-11T13:49:18Z")

</div>

Are you using log rotation around midnight? What kind of log ration are you using? What happens to the rotated files?

---

<div class="post-metadata">

**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\
**Post date:** [February 11, 2016, 1:52pm UTC](https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487/3 "2016-02-11T13:52:21Z")

</div>

It was strange , there was no log rotation at those servers where log was missed .

also my architecture is something like this -

But for general information what will be impact of log rotation we generally use logrotate feature of linux and how to overcome from these kind of failures

filebeat -\> shipper -\> kafka \<- indexer -\> elastic \<- kibana

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 11, 2016, 1:57pm UTC](https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487/4 "2016-02-11T13:57:21Z")

</div>

If you use the general log rotation there shouldn't be an issue. Filebeat finishes reading the old file and picks up the new one.

I assume shipper and index above are both LS instances. Any chance to share the log files of filebeat around the time the events went missing? Does it happen every day or it happened only once? Which version of the following are you using?

- Filebeat
- Logstash
- beat-input-plugin

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:55pm UTC](https://discuss.elastic.co/t/filebeat-missing-some-log-lines/41487/5 "2017-07-05T21:55:46Z")

</div>