Recommendations for parsing 1000's ~10MB files to backfill elasticsearch

What is the ratio between lines published and lines filtered out. The registry file keeps track of the file offset, but needs some IO to be written. If the ratio is somewhat 'bad', then the registry writes will slow down filebeat, as it also requires some fsync when writing the registry. Setting filebeat.registry_flush: 1s helps in this case (See registry_flush docs).