How many files are you writing? One per day, 10 per hour?
I suggest that you use a different input.
Perhaps you could use python to write to Redis and use the Redis input to read the CSV strings.
For a more robust solution migrate to Kafka if necessary.
The Logstash file input was not designed to work for your use case.
Where/what is your python script getting the CSV data from?