Logstash does not parse all logs with file as input

[zaeemmasood]

Hi All. I am using file input plugin to read logs. My config looks like below:

input {
  file {
    path => "/opt/gtal/iptal/elasticsearch/app/logstash/stage/CVMQA_UAT_STATS-*.txt"
    codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }

I noticed that new file does get read every time but for some reason it reports "no change". Trace was enabled and following is seen:

[2021-06-15T15:09:47,095][TRACE][filewatch.tailmode.processor][main] Active - no change {"watched_file"=>"<FileWatch::WatchedFile: @filename='CVMQA_UAT_STATS-15062021.txt', @state='active', @recent_states='[:watched, :watched, :ignored, :watched]', @bytes_read='7469', @bytes_unread='0', current_size='7469', last_stat_size='7469', file_open?='true', @initial=false, @sincedb_key='1613141125 0 64770'>"}

This report from filewatch seems untrue as logstash seems to be reading file with this filesize for the first time.

Is there a way to "force" logstash to read/ parse every file regardless of filesize or inode value?

[Badger]

@zaeemmasood

Is there a way to "force" logstash to read/ parse every file regardless of filesize or inode value?

Not with a file input. Finding out definitively whether a file has been read would require recording a checksum or hash of the contents, and recalculating that each time the file is looked at. It would be really, really expensive. The file input uses a vastly cheaper shortcut that gets things wrong from time to time.

To be sure of reading things you would have to manage that outside of logstash and use another input. curl into an http input perhaps.