Logstash sending the complete file when new data is added to the log file

Elastic Stack Logstash
1

Hi,

Given the log file (apache.log) as input to Logstash, after applying some filter the output data is stored in Elasticsearch. Here, whenever the new data is added to the log file, logstash is sending the complete file to elasticsearch instead of updating the new events.

For example, for the first time apache.log file contains 10 lines of data, after parsing this file using logstash i have verified the count in elasticsearch. It's showed me docs.count is "10". After sometime, there are 5 new lines added to the apache.log file, logstash started sending events to elasticsearch. This time i have verified the count again, it's showing the docs.count is "25". I think the docs.count should be 15, not sure.

Any ideas/suggestions would be helpful.

The logstash configuration file is below:

input {
    file {
        path => "/path/to/logfile/"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    elasticsearch {}
    stdout {}
}
2

That may be why.
When the files is updated, it's not creating a new file with the same name is it?

3

Another possible reason is that 'file' input plugin doesn't manage to store sincedb file.
Do you see a file named like ~/.sincedb ?
If not, I advice you to explicitly set sincedb_path => "/path/to/sincedb"

4

The same log file is getting updated with new data. It's not creating another file with same name.

5

There is a since_db file, which is created under home directory (~/.sincedb_514abc). It has some values inside the file, for three columns.

6

@Sri_ram If you have resolved this, please put the solution in here. It may help someone in future :slight_smile:

7

It was problem with vi editor. Whenever i tried to add data manually in to the file using vi editor, some editors creates new file instead of adding the data in to existing file. By using the command echo 'message' >> filename, i have added the data in to the log file. After that Logstash is sending only the updated events.

1 Like