I am trying to send log files from logstash to elasticsearch.
I have completed the whole setup and the test to send the standard input to elasticsearch has passed.
Now, I want to send the file to elasticsearch by specifying the file, but nothing is sent to elasticsearch.
Logstash seems to be working fine.
(At least it's not outputting any errors at runtime.
What do I need to check and fix to send the file to elasticsearch?
In the above sample, I used an "already completed" file.
However, it is not complete because what I want to put into practical use in the future is the log file.
I have tried the two cases you gave me.
mode => "tail"
start_position => "beginning"
But in both cases, the result is the same: no logs are sent to elasticsearch.
Do you have any other ideas?
You are right! I heard very good things!
I specified the sincedb_path as you suggested and confirmed that it is registered in elasticsearch.
If I want to benefit from sincedb, what should I specify for sincedb_path?
If I am running logstash for the first time, I don't think sincedb exists, do you?
sincedb will be automatically created in the right place if you do not set it ... The sincedb_path will be shown in the Logstash logs during startup, you can set it if you want but really no need.
If I recall the sincedb is created upon starting Logstash the first time.
Remember /dev/null is really for dev and testing .
I kind of understood that sincedb is automatically created in the right place.
When I searched for sincedb, I found that in my environment /usr/share/logstash/data/plugins/inputs/file/.sincedb__******************
In my environment, there are several files called "sincedb".
I think some of them are from rewriting the config and running logstash several times.
However, this means that when I clean up sincedb to reload the files
I don't know which ones to delete.
Is there any way to determine these?
Or is it possible to name the sincedb for each type of log?
For example, if I want to run nginx, apache sincedb_nginx, sincedb_apache or nginx/sincedb__****** apache/sincedb_******
, etc.
Yes, as described in the documentation the config setting sincedb_pathpoints to a file path for this specific file input sincedb. So, setting sincedb_path to <your directory>/nginx/sincedb would work.
Otherwise, as @stephenb has pointed out you do not necessarily need to delete the sincedb. You could temporarily point your sincedb_path to /dev/null for reading the existing entries. after this is done you can reset this config change and restart LogStash. After this, LogStash will read all new entries as expected.
I have confirmed that sincedb can be placed in any directory.
I'll specify /dev/null during the test of file reading, and create sincedb in the given path during operation.
Finally, I will post the logstash-sample.conf that I created.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.