@Mustapha_MJ
One possibility is that this pipeline already ran at one point, thereby generating a sincedb file. That would lead each subsequent run against the same file to conclude that the data had already been processed.
Try doing something like:
sincedb_path => "/dev/null"
OR
#for Windows
sincedb_path => "NUL"
That will disable the sincedb mechanism and should reprocess the whole file (since you've added start_position: beginning) each time you run it.
Also, if you're on version 6.4+, check out the new read mode which might suite your use case.
try removing the filter {} and changing output to just stdout - see if you can get something written to the console
try changing input to stdin in combination with the above stdout and do something like type your-input-file.txt | logstash -f your-pipeline.cof - see if you can get something written to the console
what i think is the problem is file input problem cuz i can read from stdin and send data to elasticsearch but cannot send data as txt file
logstash cannot end input file, after the final line in the file it stuck that is why it stop at : [Successfully started Logstash API endpoint {:port=>9600}”] and never send the data not exit automatically.
so how to inform logstash that the EOF is when you finish the lines ????
Logstash is meant to be a service that continuously processes data, so there is not a built-in way to stop it after processing a file. It is ok to kill it manually or via a script.
There are a bunch of other posts covering it if you want to dig in more:
Forgot to mention, if stdin{} is working, then there is probably something wrong with your path setting in file{} (did you switch your backslashes to forward slashes?) or the file is unreachable in some way (permissions, maybe?)
it turns out that logstash service doesn't feel the changes in the config file unless you change its name, after rename the config file i can finally ingest my logs into elastic search
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.