# File beat reading data from rotated File not in prospect

**URL:** <https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 14, 2016, 2:24pm UTC](https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400 "2016-04-14T14:24:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rachit\_Puri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rachit_puri/32/5721_2.png) [@Rachit\_Puri](https://discuss.elastic.co/u/Rachit_Puri)\
**Post date:** [April 14, 2016, 2:24pm UTC](https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400/1 "2016-04-14T14:24:44Z")

</div>

I am using two filebeat instance to dump logs from my application server. One dumps the accessLog and the other dumps the tomcat logs. My pipeline is Filebeat - Logstash - ElasticSearch. Application logs have rotation policy of 50 MB and it takes around 30 minutes to an hour for the rotation. I am running exactly the similar configuration on my 25 application servers. Few of them are behaving in a weird manner.

They are dumping logs from a rotated file. On deeper investigation of the registry file, i grep the inode of the file and could trace it to bear a rotated file reference  
`/usr/local/tomcat/logs/2016-04/oAuthLogs-04-14-2016-13`

Surprisingly, when i grep the inode number from registry file after 5 minutes, i could see the reference to  
`/usr/local/tomcat/logs/2016-04/oAuthLogs-04-14-2016-11`

Though, logs are being constantly being dumped in oAuthLogs file. This behavior is happening on frequent basis for some servers

filebeat version 1.1.0 (amd64)

```
filebeat:
 # List of prospectors to fetch data.
prospectors:
-
  paths:
    - /usr/local/tomcat/logs/oAuthEventLogs.log
    - /usr/local/tomcat/logs/roleCheck.log
    - /usr/local/tomcat/logs/oAuthLogs.log
    - /usr/local/tomcat/logs/oAuthBaseListenerLogs.log

  input_type: log
  ignore_older: 10h
  scan_frequency: 10s
  harvester_buffer_size: 32384
  multiline:
    pattern: "201*"
    negate: true
    match: after
  spool_size: 2000
  registry_file: /var/lib/filebeat/registry2

output:
  logstash:
   hosts: ["ipAddress:5043"]
   worker: 1
   index: index

```

File beat configuration for second instance

```
filebeat:
 prospectors:
  -
  paths:
   -/usr/local/tomcat/logs/localhost_access_log.txt

  input_type: access
  ignore_older: 10h
  scan_frequency: 10s
  harvester_buffer_size: 16384
  spool_size: 1000
  registry_file: /var/lib/filebeat/registry3

output:
 logstash:   
  hosts: ["ipAddress:5042"]
  worker: 1

```

Can, some one guide me what's happening wrong here.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [April 15, 2016, 6:27am UTC](https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400/2 "2016-04-15T06:27:03Z")

</div>

This could be related to the issue here:

- [https://github.com/elastic/beats/issues/1281](https://github.com/elastic/beats/issues/1281)
- [https://github.com/elastic/beats/pull/1375](https://github.com/elastic/beats/pull/1375)

---

<div class="post-metadata">

**Author:** ![Rachit\_Puri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rachit_puri/32/5721_2.png) [@Rachit\_Puri](https://discuss.elastic.co/u/Rachit_Puri)\
**Post date:** [April 15, 2016, 10:22am UTC](https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400/3 "2016-04-15T10:22:48Z")

</div>

Thanks, for the prompt reply.  
Will update my version and let you know

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [April 15, 2016, 10:38am UTC](https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400/4 "2016-04-15T10:38:39Z")

</div>

@Rachit_Puri It is not merged yet ...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:53pm UTC](https://discuss.elastic.co/t/file-beat-reading-data-from-rotated-file-not-in-prospect/47400/5 "2017-07-05T21:53:13Z")

</div>