Currently we are having an issue while creating a new process to send data from SQL table to Kafka. We created 2 SQL agent jobs, each to create(replaces same file daily) the a file daily on our server which filebeat need to pick and send to Kafka. Most of the flow is working as expected but for one issue. SQL agent job is failing as the file is held by FIlebeat. These jobs are completing fine if the filebeat is in stopped state. Tried close_eof, close_inactive with no luck. We are using 6.4.2 version. Could anyone help us on this issue
P.S. As of now the only solution we can think of after multiple tests is to stop and start the filebeat daily during these jobs run using windows task schedular.