Netflow beats not completing index'ing resulting in ILM not deleting last data indice of the day

Hello,

We are running filebeat's with netflow extention to keep track of data. We are running into the issue that the last incomplete indice is not completed/finished and there for never deleted by ILM.

This results in:
ILM removes .ds-netflow-2024.01.21-2024.01.21-000001 green open 1 1 95603480 101.7gb
ILM removes .ds-netflow-2024.01.21-2024.01.21-000002 green open 1 1 94638992 100.55gb
ILM will not remove .ds-netflow-2024.01.21-2024.01.21-000003 green open 1 1 18444283 19.7gb
ILM removes .ds-netflow-2024.01.22-2024.01.22-000001 green open 1 1 94638992 100.55gb

 .ds-netflow-2024.01.21-2024.01.21-000001
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "true"
  },

 .ds-netflow-2024.01.21-2024.01.21-000003
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "false"
  },

 .ds-netflow-2024.01.22-2024.01.22-000001
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "true"
  },

The result being that over time the disk will fill up with the last indice of the day. My question is: How can I set beats to set indexing_complete / close / finish when it roll's over to the next day.

Sadly no solution was found. We disabled LCM and moved it to a python script.

save_indice_for = 20 #days
double_check_if_removed = 7 #days after save_indice_for


print(f"[10/10] Removing old indices to save on storage.")

today = datetime.now().date()
date_format = "%Y.%m.%d"

for i in range(double_check_if_removed):
    current_date = today - timedelta(days=i+save_indice_for)
    data_stream_name = "netflow-" + current_date.strftime(date_format)
    print("attempting to remove data_stream: ",data_stream_name)
    es.indices.delete_data_stream(name=data_stream_name, ignore=[401, 404])