I plan to use ELK stack for analytics of my application logs.
My application runs on openstack environment which has different VMs handling application logic.
The application logs and counters from each of the VMs is already being stored at a management VM every one hour. These tar zipped logs are segregated by folders with hostnames of VMs.
So, using logstash I want to run some patterns on these logs generated every one hour and show on the kibana UI.
How can I make logstash work on these folders named with VM hostnames on latest 5 logs every one hour?
I used grok to parse a pattern and show on kibana, but I am not sure how I can run separate folders so that visualisation in kibana is folder wise.