Logstash behaviour is inconsistent

Elastic Stack Logstash
1

Hi,

I have recently started learning logstash by following the tutorials. Currently I am facing a weird problem.
I have my config as below,

input {
    file {
        path => "/home/test/project/logstash_sample_data/logstash-tutorial.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    file {
        path => "/home/test/project/logstash_sample_data/output.txt"
    }
}

Commnd to run:
./logstash -f ../../logstash_conf/first-pipeline.conf
First time when I run with this config, I got the output in output.txt. But after clearing output.txt, once again if i try, I dont get any output. I could not exactly guess the pattern that when it works correctly or when its not. I have been struggling with this problem for the past 3 days.
Do we need to clear any cache or any setting before processing the same input file again and again ? I have no clue at the moment. Could someone help me out to solve this issue ?

2

With the file input, you're likely hitting the sincedb, which remembers the last position in the file that was read.

Even with start_position => beginning, the sincedb will prevent re-reading the file from the beginning on the next run. It will try to resume where it left off.

If you plan on re-reading the same file, I suggest specifying the sincedb path manually and then deleting that file between runs.

1 Like
3

Another alternative it to cat/type the file and pipe it into a stdin inout.

4

Thanks @theuntergeek. Your technique worked for me. Now I clear the since_db file everytime and I see the output.

5

thanks @warkolm. Will try this method.