How to ignore previous log files when using file input plugin?

Elastic Stack Logstash
1

Hi guys, I am a newbie in logstash. I am trying to forward all my log files to elasticsearch using logstash. The problem is, the previously existing log files are huge and I do not need them. Is there a way to only input the newly generated log files?

To make it clear, I did tried to set the start_position => "end", and sincedb_path => "/dev/null". It did not work. Hope you guys can help me figure it out.

Thanks in advance.

2

First of all, don't set sincedb_path to /dev/null. Logstash does a pretty good job at keeping track of where it left off. Don't sabotage that.

With start_position => "end" Logstash should under no circumstances process a file from the beginning. Please show your configuration. Also, increasing logging verbosity by starting Logstash with --verbose or --debug should give hints about what Logstash is doing.

3

You can take a look at the exclude configuration option. So you can
exclude the existing files from being processed.

more info here: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-exclude

4

Hi Magnus,

If you don't recommend to set sincedb_path to "dev/null", How can I be able to read the file of the beginning?,

5

If you don't recommend to set sincedb_path to "dev/null",

Not under the circumstances described above...

How can I be able to read the file of the beginning?,

...but your circumstances are the exact opposite.