Filebeat injest same file repeatedly on an interval for logstash filter testing / development

Elastic Stack Beats
1

Hello,

I am looking to cut the cycle time when developing and testing Logstash configuration file code with a filebeat input.

I am working on grok patterns, filters, etc and my current test cycle involved re-ingesting sample files with filebeat and observing the Logstash stdout. This is fine, but I had to clear the filebeat data registry to reingest the same file, and I had to stop and start filebeat.

I would like my testing cycle to be:

  1. Filebeat injesting same sample log in N interval,
  2. logstash config auto updating, emitting to stdout

#2 is good. I have not figured out how to get FB to injest the same file on N interval.

I tried some filebeat mocking w/ the generator and stdin input filters. Neither were good. The generator just filled up the stdout because I was unable to "emit line on N interval". I tried a variety of stuff w/ stdin and could not get a basic string to logstash's stdout.

I am running the ELK stack @ 6.2.1.

ideally, I would periodically like to injest one sample log file in filebeat on N interval. I am relatively new to this stack, so it's not clicking yet.

It looks like scan_frequency will set the N interval, but I am still unable to find the prospector setting which will reingest the same file. I have a feeling it may be the clean and ignore_older settings?

2

If you are not committed to filebeat then you could use an exec input in logstash that runs cat every "interval" seconds.

3

Filebeat doesn't have a facility to re-injest a file after a predefined interval.

However, you can trick it to believe it's a new file, if you delete the log file and copy it from an external location.

For example if you're watching /var/log/*.log, every N interval you can do:

rm /var/log/watched.log
cp source_dir/sample.log /var/log/watched.log

With this the file should be getting a new inode number everytime so filebeat is tricked into believe is a new file.

4

This topic was automatically closed after 21 days. New replies are no longer allowed.