Apologies if this has been asked, I did search but couldn't find this issue.
I am currently testing a build of the ELK stack to monitor some of our servers. I have decided to use Beats for the log shipping due to awful nature of the multiple logs I need to ship.
I configured Logstash and ElasticSearch with Kibana and all are up and running.
I have installed Beats on one of the servers and when first set up it worked fine and scanned the files, these were then sent to Logstash and onto Elasticsearch.
Due to the testing process I am trying to sort out the timestamp and therefore deleted the indices from Elasticsearch with the idea to rescan.
I need to recreate the index on Elasticsearch but Beats is now refusing to send any data to Logstash. Even if I copy new unscanned log files into the prospector directory it sends nothing and no new index is created. Surely new files (read unscanned) should be parsed?
I have stopped and restarted Beats and can see through Process Monitor that Beats is indeed scanning the files.
What am I missing? The issue seems to be with Beats deciding whether or not (not in this case) to send the data from the parsed log files.
Any help would be appreciated.