File Diff Beat or File Meta Beat

[pat_hougi (Pat Hourigan)]

Hi There,

I have a microService that computes diffs for config files, stores it off to a graphDB and does all sorts versioning and other fun stuff.

I want to see if it's possible to use Beats and Logstash to invoke this micrService when a configuration file changes on disk.

To this end, I was hoping there would be a Beat to help. I'd like the Beat to watch a series of files (config XML, properties, etc) and when the file is modified or a new one is added or deleted to a dir, the Beat will fwd the meta data about the file being modified/deleted/added to Logstash along with a full stream of the file, which in turn will be sent to my microService.

I've had a brief look at the ExecBeat and the FileBeat, but neither are fully compatible, the FileBeat is more suited to parsing log files.

any thoughts on how this could be achieved, or should we be looking at writing a new Beat called FileDiffBeat or DirectoryBeat ?

[ruflin]

I remember a request for something like this popped up in the past but more from a security perspective. I would definitively be possible to build a beat that only watches a directory for files to change and then sends an event.

We also recently added the audit module to metricbeat. Not sure if you could configured auditd somehow that you get a notification on file changes only: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-metricset-audit-kernel.html