Hi,
I am in a development/testing phase of implementing the ELK stack. So I play and modify patterns and would like to see in Kibana the result of those changes as soon as possible. On my system, I am trying to test a multiline message from an exception file.
My setup is the following:
Filebeat -> Logstash -> Elasticsearch ->Kibana
Filebeat has a prospector that already detects a multiline event and aggregates it. Logstash does some filtering and adds some extra tags and then everything is sent to Elasticsearch and I can search for them in Kibana.
When I modify the multiline pattern in Filebeat and restart the service, there is no option to reparse the file from the beginning like for regular files on local machine( sincedb or start_position). Even if I restart the Logstash and delete the indexes from Elastic search these lines do not appear. I am guessing that Filebeat or Logstash keeps a record of the last position that was sent from a specific file and starts from there. Even deleting the prospector and then restarting the service and re-adding the prospector will not reparse the file.
Filebeat creates a file which keeps a record of the last position. The parameter is registry_file, you can set in any path you want create it (in the filebeat.yml), example:
filebeat.registry_file: /home/user/filebeat/data
Moreover, there is another parameter, it is ignore_older, if you want to parse old logs, you should set this parameter.
I guess this is an optional value. The default value should point to {path.home}/data and in my case - with a default installation - this folder does not even exists. That is strange because the last position value has to be stored somewhere.
The ignore_older option is exactly the contrary of what I am trying to accomplish. I want to get all data again from beginning, even if has already been sent and even if it will (potentially) create duplicates.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
, I have just seen in the documentation a section which talks about the default paths: