I'm new to logstash so forgive any obvious mistakes! I've recently installed the ELK stack and have been trying to setup a pipeline to elasticsearch but have hit a wall. My config file (see below) seems to run without a hitch but for some reason my CSV isn't being logged - I've checked the structure of my CSV and edited it recently so there doesn't seem to be a problem there.
This behaviour is due to the strict way that the "tailing" concept has been interpreted, like the unix tail -f command.
If a file is present when Logstash starts, then reading starts from the end, i.e. only process new content that is seen in the file after LS starts. The start_position is meant for when a file is "discovered" after LS starts where one can override this tail from the end.
The version of the file input that is shipped with LS 6.4.0 has a "read" mode - specifically designed for when a file should be read from start to finish (subject to sincedb position tracking, which you opted out of by setting the sincedb_path to NUL).
Along with read mode is the ability to do some action when the EOF is reached. The default action is to delete the file, but you can change this to "log" as well as adding a path to a file that will hold the path to each file as it is completed. If using the delete option, make sure that you have a copy of the original.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.