Netflow beats not completing index'ing resulting in ILM not deleting last data indice of the day

[eddie4]

Hello,

We are running filebeat's with netflow extention to keep track of data. We are running into the issue that the last incomplete indice is not completed/finished and there for never deleted by ILM.

This results in:
ILM removes .ds-netflow-2024.01.21-2024.01.21-000001 green open 1 1 95603480 101.7gb
ILM removes .ds-netflow-2024.01.21-2024.01.21-000002 green open 1 1 94638992 100.55gb
ILM will not remove .ds-netflow-2024.01.21-2024.01.21-000003 green open 1 1 18444283 19.7gb
ILM removes .ds-netflow-2024.01.22-2024.01.22-000001 green open 1 1 94638992 100.55gb

 .ds-netflow-2024.01.21-2024.01.21-000001
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "true"
  },

 .ds-netflow-2024.01.21-2024.01.21-000003
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "false"
  },

 .ds-netflow-2024.01.22-2024.01.22-000001
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "true"
  },

The result being that over time the disk will fill up with the last indice of the day. My question is: How can I set beats to set indexing_complete / close / finish when it roll's over to the next day.

[eddie4]

Sadly no solution was found. We disabled LCM and moved it to a python script.

save_indice_for = 20 #days
double_check_if_removed = 7 #days after save_indice_for


print(f"[10/10] Removing old indices to save on storage.")

today = datetime.now().date()
date_format = "%Y.%m.%d"

for i in range(double_check_if_removed):
    current_date = today - timedelta(days=i+save_indice_for)
    data_stream_name = "netflow-" + current_date.strftime(date_format)
    print("attempting to remove data_stream: ",data_stream_name)
    es.indices.delete_data_stream(name=data_stream_name, ignore=[401, 404])