# Netflow beats not completing index'ing resulting in ILM not deleting last data indice of the day

**URL:** <https://discuss.elastic.co/t/netflow-beats-not-completing-indexing-resulting-in-ilm-not-deleting-last-data-indice-of-the-day/352851>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 8, 2024, 11:13am UTC](https://discuss.elastic.co/t/netflow-beats-not-completing-indexing-resulting-in-ilm-not-deleting-last-data-indice-of-the-day/352851 "2024-02-08T11:13:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![eddie4](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@eddie4](https://discuss.elastic.co/u/eddie4)\
**Post date:** [February 8, 2024, 11:13am UTC](https://discuss.elastic.co/t/netflow-beats-not-completing-indexing-resulting-in-ilm-not-deleting-last-data-indice-of-the-day/352851/1 "2024-02-08T11:13:11Z")

</div>

Hello,

We are running filebeat's with netflow extention to keep track of data. We are running into the issue that the last incomplete indice is not completed/finished and there for never deleted by ILM.

This results in:  
ILM removes .ds-netflow-2024.01.21-2024.01.21-000001 green open 1 1 95603480 101.7gb  
ILM removes .ds-netflow-2024.01.21-2024.01.21-000002 green open 1 1 94638992 100.55gb  
ILM **will not** remove .ds-netflow-2024.01.21-2024.01.21-000003 green open 1 1 18444283 19.7gb  
ILM removes .ds-netflow-2024.01.22-2024.01.22-000001 green open 1 1 94638992 100.55gb

```
 .ds-netflow-2024.01.21-2024.01.21-000001
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "true"
  },

 .ds-netflow-2024.01.21-2024.01.21-000003
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "false"
  },

 .ds-netflow-2024.01.22-2024.01.22-000001
  "lifecycle": {
    "name": "filebeat",
    "indexing_complete": "true"
  },

```

The result being that over time the disk will fill up with the last indice of the day. My question is: How can I set beats to set indexing\_complete / close / finish when it roll's over to the next day.

---

<div class="post-metadata">

**Author:** ![eddie4](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@eddie4](https://discuss.elastic.co/u/eddie4)\
**Post date:** [February 15, 2024, 12:49pm UTC](https://discuss.elastic.co/t/netflow-beats-not-completing-indexing-resulting-in-ilm-not-deleting-last-data-indice-of-the-day/352851/2 "2024-02-15T12:49:42Z")

</div>

Sadly no solution was found. We disabled LCM and moved it to a python script.

```auto
save_indice_for = 20 #days
double_check_if_removed = 7 #days after save_indice_for

print(f"[10/10] Removing old indices to save on storage.")

today = datetime.now().date()
date_format = "%Y.%m.%d"

for i in range(double_check_if_removed):
    current_date = today - timedelta(days=i+save_indice_for)
    data_stream_name = "netflow-" + current_date.strftime(date_format)
    print("attempting to remove data_stream: ",data_stream_name)
    es.indices.delete_data_stream(name=data_stream_name, ignore=[401, 404])

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 14, 2024, 2:50pm UTC](https://discuss.elastic.co/t/netflow-beats-not-completing-indexing-resulting-in-ilm-not-deleting-last-data-indice-of-the-day/352851/3 "2024-03-14T14:50:15Z")

</div>

Closed on March 14, 2024, 2:50pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.