# Filebeat not reading logs from subdirectories

**URL:** <https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905>\
**Category:** Logstash\
**Created:** [June 10, 2022, 4:17pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905 "2022-06-10T16:17:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![garry12](https://avatars.discourse-cdn.com/v4/letter/g/ac8455/32.png) [@garry12](https://discuss.elastic.co/u/garry12)\
**Post date:** [June 10, 2022, 4:17pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905/1 "2022-06-10T16:17:39Z")

</div>

Hey team, I am relatively new to ELK stack and I am trying to send logs from a linux servers to Elasticsearch. The path I am choosing is -  
I have installed the filebeat on linux server where my application logs are getting generated - \> parsing them via logstash and then - \> sending them to elasticsearchQuestions I have are -

1. The linux server having application logs generates and stored logs in directories dynamically based on what day/month/time of the day its running for example my directory structure for logs on 06/10/2022 at 11:45 am will look like --/var/log/2022/06/10/11/abc.txt

I want my filebeat input path for logs in filebeat.yml to take paths dynamically so that I do not have to keep changing the paths and restarting the filebeat service so I tried to use something like - /var/log/2022/\*/\*/\*/\*.txt

But when I specify file path with wildcards like /var/log/2022/\*/\*/\*/\*.txt I get no logs shipped and the filebeat service runs fine but harvester always shows 0 files and no logs gets shipped however when I changes that to specifically point to any folder without using wildcard like -/var/log/2022/06/10/11/abc.txt the logs get shipped and I can see them in Elasticsearch. So, I want to know what I should do in order to make this dynamic path work and the filebeat version I am using is 7.17.0.  
Please let me know if you guys have any ideas.

(Note: - There are 12 folders inside 2022 for months like - 01, 02, 03 etc ..

Inside those folders for months there are sub folders to support dates depending on how many days in the months like - 01, 02...29,30 ...etc\*

Inside those there are 23 subfolders for hours of the day like - 00,01,02...23 )

1. Another question I have is whenever I get the logs to get shipped I see latency like I ideally want logs to immediately appear in Elasticsearch as soon as they appear in linux server where the application is running and generating the logs but I always see a latency like logs appears in Elasticsearch at least with a delay of 5-15 mins so how can I make it appear as soon as they show up like real time ?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 10, 2022, 6:38pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905/2 "2022-06-10T18:38:32Z")

</div>

Please share your `filebeat.yml` file using the Preformatted text button, `</>`.

---

<div class="post-metadata">

**Author:** ![garry12](https://avatars.discourse-cdn.com/v4/letter/g/ac8455/32.png) [@garry12](https://discuss.elastic.co/u/garry12)\
**Post date:** [June 10, 2022, 6:41pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905/3 "2022-06-10T18:41:06Z")

</div>

```auto
----This does not works ---
filebeat.inputs:
- input_type: log
  enabled: true
  paths:
    - /.pss_filecore/2022/*/*/*/*.txt
  multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 3500

output.logstash:
  hosts: ["10.10.10.113:5044"]

```

---

<div class="post-metadata">

**Author:** ![garry12](https://avatars.discourse-cdn.com/v4/letter/g/ac8455/32.png) [@garry12](https://discuss.elastic.co/u/garry12)\
**Post date:** [June 10, 2022, 6:42pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905/4 "2022-06-10T18:42:03Z")

</div>

```auto
----But this works ---
filebeat.inputs:
- input_type: log
  enabled: true
  paths:
    - /.pss_filecore/2022/06/10/14/*.txt
  multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 3500

output.logstash:
  hosts: ["10.10.10.113:5044"]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2022, 6:42pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-logs-from-subdirectories/306905/5 "2022-07-08T18:42:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.