File input plugin lost first part of log line


(Juan Alfredo Rella) #1

We're working with the version 2.3.x of logstash and we've defined a input file plugin to read apache logs on this way

file {
	add_field => ["timestamp", ""]
	path => "/var/log/httpd/access_log.*"
	start_position => end
	type => "beaconEvent"
	sincedb_path => "/since/.sincedb*"
	sincedb_write_interval => 15
}

we've detected that sometimes this plugin reads an incomplete access log line. It lose an unknown amount of characters at the beginning of this line.
It seems to be related with the offset it is handle on sincedb_path but we could not find a pattern. we removed all files of sincedb_path it worked ok with a big volume or records.
We've watched there was a same issue on previous versions of this tool (https://logstash.jira.com/browse/LOGSTASH-1503) . does anybody could help me?


(Lionel Cons) #2

We have seen the same problem with Kafka logs (log4j) and Logstash 5.4.

This might be the same problem as https://github.com/logstash-plugins/logstash-input-file/issues/151 (Lost data on rotation).