I need real time data monitoring in such a way that all existing as well as new csv files should be monitored. Both files should get monitored from beginning and any monitored files if are modified in future should get monitored from the end.
Why are you starting a new thread?
What you describe is exactly how the file input works. Your problem is rather hard to debug remotely. I suggest you increase the log level with --verbose or --debug to get clues about what's happening.
How are you determining that no data is being picked up?
This thread is concerned only with "end" in start_position.
I tried --verbose but it stop with message "Pipeline started".