# How does filebeat traverse a path

**URL:** <https://discuss.elastic.co/t/how-does-filebeat-traverse-a-path/99038>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 31, 2017, 6:39pm UTC](https://discuss.elastic.co/t/how-does-filebeat-traverse-a-path/99038 "2017-08-31T18:39:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hburnswell](https://avatars.discourse-cdn.com/v4/letter/h/9e8a1a/32.png) [@hburnswell](https://discuss.elastic.co/u/hburnswell)\
**Post date:** [August 31, 2017, 6:39pm UTC](https://discuss.elastic.co/t/how-does-filebeat-traverse-a-path/99038/1 "2017-08-31T18:39:43Z")

</div>

Hi All,

I'm curious about how filebeat behaves on a given path. My goal is to ingest IIS logs as filebeat -\> logstash -\> elasticsearch. The IIS log directory has log files going back to Jan 2016 (1 per day). I am using 'ignore\_older' in my filebeat.yml file to ingest approximately the last 2 months of log information as:

> ignore\_older: 1344h

1. I was unable to use 'd' as a directive as it said it was 'unknown'. Does ignore\_older just accept 'h' or 's'?
2. In Kibana, when I filter the index on log\_timestamp, the oldest date I see is: 2017-08-13. Can I assume that the ingest of logs has not yet brought in any logs older than that date?
3. Is there a better/more efficient way to accomplish what I'm looking to do?

I am using the following in my filebeat.yml file:

> ```
> paths:
> - C:\path\to\files\file_pattern*
> 
> input_type: log
> ignore_older: 1344h
> 
> ```

1. Is using the 'file\_pattern\*' causing any inefficiency? Should I just use \path\to\file\dir if all the files in the directory are specific to what I want to ingest?
2. How does filebeat traverse the designated path? Does it start with the oldest file timestamp and work it's way to the newer files (this would seem not to be the case given the above date filtering on log\_timestamp)?
3. Does filebeat skip files that have an older time stamp than the designated ignore\_older directive or does it still dow some sort of parsing?

I am happy to read any documentation I can be pointed to. I appreciate any guidance.

Thanks,

HB

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [September 1, 2017, 8:18pm UTC](https://discuss.elastic.co/t/how-does-filebeat-traverse-a-path/99038/2 "2017-09-01T20:18:49Z")

</div>

> [@hburnswell](#):
>
> I was unable to use ‘d’ as a directive as it said it was ‘unknown’. Does ignore\_older just accept ‘h’ or ‘s’?

Valid time units are ns, us, ms, s, m, h. [Config file data types | Beats Platform Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format-type.html#_duration)

> [@hburnswell](#):
>
> In Kibana, when I filter the index on log\_timestamp, the oldest date I see is: 2017-08-13. Can I assume that the ingest of logs has not yet brought in any logs older than that date?

Are you parsing your logs with Logstash in order to extract the date from the log line into `log_timestamp`? If so I'd say the answer is yes assuming you using a large time range in Kibana when viewing the data.

> [@hburnswell](#):
>
> Is there a better/more efficient way to accomplish what I’m looking to do?

You are using `ignore_older` correctly.

> [@hburnswell](#):
>
> Is using the ‘file\_pattern\*’ causing any inefficiency? Should I just use \path\to\file\dir if all the files in the directory are specific to what I want to ingest?

The difference is probably negligible. But if all the files in the dir match `file_pattern*` I would drop the file name pattern from the glob leaving just `'C:\path\to\files'`.

> [@hburnswell](#):
>
> How does filebeat traverse the designated path? Does it start with the oldest file timestamp and work it’s way to the newer files (this would seem not to be the case given the above date filtering on log\_timestamp)?

It asks the OS for a directory listing and starts harvestors in the order of the listing provided by the OS. All of the file havestors run concurrently.

> [@hburnswell](#):
>
> Does filebeat skip files that have an older time stamp than the designated ignore\_older directive or does it still dow some sort of parsing?

When filebeat gets the directory listing it checks each file's last modified time. If that time is older than the `ignore_older` period then no harvestor is started for that file (so it never reads any file content).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 30, 2017, 8:41pm UTC](https://discuss.elastic.co/t/how-does-filebeat-traverse-a-path/99038/4 "2017-09-30T20:41:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.