I am looking to cut the cycle time when developing and testing Logstash configuration file code with a filebeat input.
I am working on grok patterns, filters, etc and my current test cycle involved re-ingesting sample files with filebeat and observing the Logstash stdout. This is fine, but I had to clear the filebeat data registry to reingest the same file, and I had to stop and start filebeat.
I would like my testing cycle to be:
- Filebeat injesting same sample log in N interval,
- logstash config auto updating, emitting to stdout
#2 is good. I have not figured out how to get FB to injest the same file on N interval.
I tried some filebeat mocking w/ the generator and stdin input filters. Neither were good. The generator just filled up the stdout because I was unable to "emit line on N interval". I tried a variety of stuff w/ stdin and could not get a basic string to logstash's stdout.
I am running the ELK stack @ 6.2.1.
ideally, I would periodically like to injest one sample log file in filebeat on N interval. I am relatively new to this stack, so it's not clicking yet.
It looks like scan_frequency will set the N interval, but I am still unable to find the prospector setting which will reingest the same file. I have a feeling it may be the clean and ignore_older settings?