File input v4.1.2 tailing - duplicate docs ingested

Elastic Stack Logstash
5

@ld_pvl

I want to work on this issue and the others you have open in one place.

  1. This issue and the read-to-eof-no-delimiter-found-in-current-chunk have the same origin in my opinion. It is a bug (I think) related how the plugin reacts to file rotations in the scheme used.

  2. The last outstanding issue you have regards the very long time it takes to move from the discover phase to the processing phase when initially discovering 500,000 files.

I will look at both of these shortly. 1 can affect other users but 2 is more applicable to your use case.
Both are extremely hard to writing failing test scenarios for though so patience is needed.

1 Like