I am in a development/testing phase of implementing the ELK stack. So I play and modify patterns and would like to see in Kibana the result of those changes as soon as possible. On my system, I am trying to test a multiline message from an exception file.
My setup is the following:
Filebeat -> Logstash -> Elasticsearch ->Kibana
Filebeat has a prospector that already detects a multiline event and aggregates it. Logstash does some filtering and adds some extra tags and then everything is sent to Elasticsearch and I can search for them in Kibana.
When I modify the multiline pattern in Filebeat and restart the service, there is no option to reparse the file from the beginning like for regular files on local machine( sincedb or start_position). Even if I restart the Logstash and delete the indexes from Elastic search these lines do not appear. I am guessing that Filebeat or Logstash keeps a record of the last position that was sent from a specific file and starts from there. Even deleting the prospector and then restarting the service and re-adding the prospector will not reparse the file.
Do you have some recommendations for me? thanks