# Filebeat harvester picks up files in random order when scanning
**URL:** https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542
**Category:** Beats
**Tags:** filebeat
**Created:** [May 20, 2017, 11:36pm UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542 "2017-05-20T23:36:59Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@menezeswayne](https://discuss.elastic.co/u/menezeswayne)
#### Post date: [May 20, 2017, 11:36pm UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/1 "2017-05-20T23:36:59Z")
I used the harvester\_limit: 1 option along with close\_inactive option to close a file after 1 min. This works for my use case. However when there are multiple older log files to be harvested, filebeat picks up the files in a random order instead of going for the oldest file first.
Has anyone faced this issue?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 22, 2017, 12:33am UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/2 "2017-05-22T00:33:23Z")
I'm not sure we provide explicit assurances around the order of processing.
It might be worth raising this as a feature request if it's important to you 🙂
---
### Author:  [@menezeswayne](https://discuss.elastic.co/u/menezeswayne)
#### Post date: [May 22, 2017, 1:14am UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/3 "2017-05-22T01:14:27Z")
I have a solution in my fork. If I raise a pull request do you mind taking
a look?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 22, 2017, 1:15pm UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/4 "2017-05-22T13:15:57Z")
Indeed we don't have any guarantees on the ordering. We discussed it in the past but didn't see the need for it as it would add scheduling complexity (as everyone wants a different order). Can you elaborate in more detail on why you need this ordering and have harvester\_limit: 1. Understanding the use cases helps a lot.
Happy to also have a look at some code.
---
### Author:  [@menezeswayne](https://discuss.elastic.co/u/menezeswayne)
#### Post date: [May 22, 2017, 10:11pm UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/5 "2017-05-22T22:11:07Z")
We have a legacy system that produces a lot of logs which rollover frequently. Our setup sends logs to logstash from where we use the http output and send it to another legacy monitoring service. It has wireless connectivity issues which means we could see gaps during which no logs get shipped over to logstash. When the connection comes back we'd like to see the log files get sent out in the same order they were created. We can tolerate some amount of re-ordering in the legacy monitoring service but in the case of filebeat the order of scan is just totally random which doesn't help at all.
---
### Author:  [@menezeswayne](https://discuss.elastic.co/u/menezeswayne)
#### Post date: [May 22, 2017, 10:14pm UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/6 "2017-05-22T22:14:13Z")
pull request here: [https://github.com/elastic/beats/pull/4374](https://github.com/elastic/beats/pull/4374)
---
### Author:  [@menezeswayne](https://discuss.elastic.co/u/menezeswayne)
#### Post date: [May 24, 2017, 1:18am UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/7 "2017-05-24T01:18:06Z")
@ruflin I have another question - how do I ensure that while cleaning up older log files that their offset isn't still tracked in the registry. I assume that when close\_inactive is reached, it gets removed from the registry?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 26, 2017, 7:02am UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/8 "2017-05-26T07:02:33Z")
Check the `clean_*` options like `clean_inactive`.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 23, 2017, 7:02am UTC](https://discuss.elastic.co/t/filebeat-harvester-picks-up-files-in-random-order-when-scanning/86542/9 "2017-06-23T07:02:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.