Filebeat not reading logs from subdirectories

Elastic Stack Logstash
1

Hey team, I am relatively new to ELK stack and I am trying to send logs from a linux servers to Elasticsearch. The path I am choosing is -
I have installed the filebeat on linux server where my application logs are getting generated - > parsing them via logstash and then - > sending them to elasticsearchQuestions I have are -

  1. The linux server having application logs generates and stored logs in directories dynamically based on what day/month/time of the day its running for example my directory structure for logs on 06/10/2022 at 11:45 am will look like --/var/log/2022/06/10/11/abc.txt

I want my filebeat input path for logs in filebeat.yml to take paths dynamically so that I do not have to keep changing the paths and restarting the filebeat service so I tried to use something like - /var/log/2022/*/*/*/*.txt

But when I specify file path with wildcards like /var/log/2022/*/*/*/*.txt I get no logs shipped and the filebeat service runs fine but harvester always shows 0 files and no logs gets shipped however when I changes that to specifically point to any folder without using wildcard like -/var/log/2022/06/10/11/abc.txt the logs get shipped and I can see them in Elasticsearch. So, I want to know what I should do in order to make this dynamic path work and the filebeat version I am using is 7.17.0.
Please let me know if you guys have any ideas.

(Note: - There are 12 folders inside 2022 for months like - 01, 02, 03 etc ..

Inside those folders for months there are sub folders to support dates depending on how many days in the months like - 01, 02...29,30 ...etc*

Inside those there are 23 subfolders for hours of the day like - 00,01,02...23 )

  1. Another question I have is whenever I get the logs to get shipped I see latency like I ideally want logs to immediately appear in Elasticsearch as soon as they appear in linux server where the application is running and generating the logs but I always see a latency like logs appears in Elasticsearch at least with a delay of 5-15 mins so how can I make it appear as soon as they show up like real time ?
2

Please share your filebeat.yml file using the Preformatted text button, </>.

3 
----This does not works ---
filebeat.inputs:
- input_type: log
  enabled: true
  paths:
    - /.pss_filecore/2022/*/*/*/*.txt
  multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 3500


output.logstash:
  hosts: ["10.10.10.113:5044"]
4 
----But this works ---
filebeat.inputs:
- input_type: log
  enabled: true
  paths:
    - /.pss_filecore/2022/06/10/14/*.txt
  multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 3500


output.logstash:
  hosts: ["10.10.10.113:5044"]
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.