Can Filebeat watch for new files created in a directory?

Elastic Stack Beats
1

Hello all,

Magento is a PHP ecommerce application. In case of severe problems, Magento crashes, generating a file for each crash in a var/report directory.
The file contains informations about the crash (db unreachable for instance). The file name is a random number.

I'm looking for a solution to detect and send those reports automatically when they appear in the var/report directory.

I tried using the directory as a path : not working:
prospector.go:293: Skipping directory: /var/www/magento/var/report/

For now I will simulate a log file with a bash script loop running in background, but this is not very handy.
I may create an enhancement issue for that.

Thanks in advance,
Marc

2

Use a wildcard: /var/www/magento/var/report/*

Note that Filebeat sends files line by line. You may want to use its multiline feature to join the lines of each file (or at least some of them) into a fewer number of events.

3

It's working, but the registry file is growing indefinitely.
Any way to disable registry state for a specific prospector ?

4

No, the registry filepath is global for the whole Filebeat instance.

5

A small drawback also: file modification date is not sent. So the event time in elastic search is the log reception date, not the report generation time.

May I open an enhancement request on GitHub ?

6

May I open an enhancement request on GitHub ?

Of course. I'd also file an issue against whatever is producing those files for not including any timestamp inside the file.

7

Here:

8

Okay, everything is fixed in 5.0.0-alpha5.

Any roadmap for the stable release ?

I doubt my hoster will accept to install the alpha release...

Thanks in advance,
Marc

9

@dooblem Glad to hear it works with filebeat 5.0. That means the efforts to rewrite large parts was worth it.

The first beta versions should come out in the next weeks. GA is planned for Q4. On the filebeat side currently no major / breaking additional changes are planned for the final version in case this helps.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.