Include Data Streams in ILM Policy

Hi There,
I am making use of Data Streams and an ILM Policy to delete data that is rolled over from Hot phase to delete phase, this happens after 3 days.

I wanted to check if there is a way to include the actual deletion of the Logstash Data Streams themselves as part of the ILM. The ILM deletes the data in the indexes but the data stream backing indexes still seem to exist, as well as the Data Streams themselves.
This pushes the index and shard count up over time.

Is there a correct way to maintain the data streams themselves and to clear out the backing indexes automatically as part of an ILM or another method?
Thanks

ILM deletes things at the index level. If you are still seeing backing indices then we'd need more information to help. Can you share the output from _cat/indices?v and an ILM explain on the policy?

Please see _cat/indices?v below

health status index                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .ds-logstash-2022.10.29-2022.11.19-000040 NAZ463lgRUKrBlSjljjgIQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.14-2022.11.21-000074 0tJbahISTgarbQ4ZvOUX3A   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.06-2022.11.22-000030 Q_kNNw7rTuCs6EgWR-3lQw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.03-2022.11.19-000030 ajgbj8ecQiWqK6L1Tgylbw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.07-2022.11.22-000028 4OKJ5yygTmmg5cPcv77N_A   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.05-2022.11.19-000026 BWPPLcDbSse2-HLR5v0NvA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.05-2022.11.22-000032 j_CdEO8rR9-cb9OZ6-BatA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.27-2022.11.18-000102 mxRhZP4qSjKivQJmnOJJ2w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.11-2022.11.18-000074 3laZdYA2TnSSB7AxWiFCnw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.31-2022.11.19-000036 EFi6A05hQe2g3fRaAt2E7Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.25-2022.11.19-000092 I5dHLWYkQ767EseqpjU6BA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.28-2022.11.19-000042 fpNx5qjJRtqVoeUB1Ln0Rw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.07-2022.11.20-000024 F8hNC3ilQVmWS9hd3gCm1Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.04-2022.11.19-000028 Us1-4GJARgCKDe7hFzawHQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.25-2022.11.20-000094 q30w-8nER8G4eZQ3xVvnJg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.10-2022.11.20-000018 RZ7uGpV-Tr-O-YKt4ZON7Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.08-2022.11.18-000080 iWGY2IHpT6KMZww0ltqPtg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.11-2022.11.20-000016 9m_SefCXTMyy83yjkwr52Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.04-2022.11.22-000034 NrpkOBHAQtKq-v-oQ0gIsg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.08-2022.11.22-000026 qIH_JqsaSgq_1lzrBauHGA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.16-2022.11.20-000100 bMEFy1aISK6rjwTpIHQO3w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.11-2022.11.21-000080 iX1yA6dxTG2uPB2hoUnnhg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.07-2022.11.18-000082 ID50efu1TMW3gwFwRTZI7w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.10-2022.11.18-000076 U7W8FPsqSlSsXKMEpLWzQQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.08-2022.11.20-000022 T8gBt-uAQ0Szn_nvz8bnjw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.16-2022.11.21-000070 4Q4Vai6bR_yxB3Ykok1o2g   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.28-2022.11.21-000100 jdvKs1xuScOHqE3wlZ4HGA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.09-2022.11.20-000020 Y9Y95zG2TZeJT8wc-kcEVg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.09-2022.11.22-000024 fiw_JkRdSl-Kdl7bjIrO3Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.12-2022.11.22-000018 8zvFbPkwSn22e6Z5drQPqQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.29-2022.11.21-000104 -imM346WQPO1Y8TWipb1qA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.15-2022.11.21-000072 vgu9pUNeRGecI2qcB8--kw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.23-2022.11.19-000092 AzbovF2oRku1GVvVHg5-cQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.14-2022.11.18-000068 4G20sHYdS0mWNMWYCvWVKQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.04-2022.11.20-000030 9oNJEsK6RDWbme1mNFatFw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.26-2022.11.19-000046 A6XDQAnhSTu-ehXGux0I4g   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.27-2022.11.21-000108 ylLYvbqJR0q6ajS7s-AEmg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.01-2022.11.20-000098 Boj0n7ScTR-FAJzdf2FNmw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.23-2022.11.20-000094 PHk_8s6cRJyoymlJlN0-5g   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.08-2022.11.19-000020 jxdpqwqATRWn9PYJYF43DQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.02-2022.11.20-000034 BaCWQ_0KTAyYjL7zMFB0kA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.06-2022.11.18-000084 33fE5Te-QFmyG2OJdFckXg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.28-2022.11.22-000048 nd_LUyiZRzaJpli7OSXgtw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.20-2022.11.21-000062 80AOzBtfQFuAxPUX0O-eOg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.30-2022.11.20-000040 FY7OAAbBS9CTq4pq3hASKw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.28-2022.11.20-000098 OYprzBpcSSC4iseSEb9wgw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.19-2022.11.21-000064 7uKP1xAiR5CuBH8blrfL7Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.15-2022.11.18-000066 WfDL4fIhQLu9nkRlWT2yIA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.29-2022.11.20-000042 YqQ_U_OuTiCE-F9xSrI5sA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.13-2022.11.20-000012 dXCPzqwaSY21sXCgVsZJEw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.03-2022.11.22-000036 hI3PQ2xGTGmR-UeG3u7GiA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.01-2022.11.22-000040 4-ACXKuHR9iRCDji39pAZA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.21-2022.11.21-000001 BC55po1oT8KYlmmiVPy8cw   3   0   35710914            0     15.7gb         15.7gb
green  open   .ds-logstash-2022.11.09-2022.11.19-000018 g1Zp4DIhSQOVonPoT5oVgQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.11-2022.11.22-000020 geVg1jhUSkKA-H8Lqsdm4w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.05-2022.11.20-000028 0UWEZ_FcS1Wg_KVh30-uog   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.04-2022.11.18-000088 qZHJJZnAQQStHiUBBARtJA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.14-2022.11.22-000014 UiSNWGmxTX2Zjog05Y2Uew   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.06-2022.11.19-000024 zmayhGntSPqTIj7IQfX33Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.30-2022.11.21-000100 9GXaX8UjR8SLWnjRsgDTGw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.23-2022.11.19-000052 tp2TyEicSpi4UsqhNJ_PCA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.12-2022.11.21-000078 8HYhJHCYSkyraxR1lNNR9w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.06-2022.11.21-000090 yr-ZC0H7Te28IvWRyGXzag   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.20-2022.11.18-000056 ksnJsAy7QWWF93V1inVGIA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.09-2022.11.21-000084 q7W7kQElT6-x8sYypyzksQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.03-2022.11.20-000094 eP8tVUJHRrWFeeuV88V4XQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.21-2022.11.20-000096 HyWbLvlKQ3Csisty8h4exg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.01-2022.11.19-000034 AZkTrpUCQCCbIz5VXZuZlA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.09-2022.11.18-000078 oZd-BsPBTZu6iBBpZ0JhOw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.19-2022.11.20-000106 dUUqJMpBSU-hOrN2v_4fGA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.17-2022.11.18-000062 gQ0bjwPrTpSxjTOCuz7fmw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.31-2022.11.22-000042 KCI7o2vsSGCFQFzns0h_Ug   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.30-2022.11.19-000096 -IeNVkYyRXi92VedwBXMnQ   3   0          0            0       678b           678b
green  open   .async-search                             C3YFN9rISJWJW4U138Nxow   1   1          0            0       528b           264b
green  open   .ds-logstash-2022.09.18-2022.11.19-000102 cUpnmn-iR7qdoHw17DN0bQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.27-2022.11.20-000046 VaJAFflNStilQGHMdO8mtg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.17-2022.11.21-000068 8vC1GTXwQLqY0mkcHiwjwQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.12-2022.11.18-000072 wN5VK1VHQRSHNRw5matLSA   3   0          0            0       678b           678b
green  open   .siem-signals-default-000001              BaHhEx-jSLCB1G_vft_iug   1   1          0            0       452b           226b
green  open   .ds-logstash-2022.10.24-2022.11.20-000052 cVBueA5ETjeEKul029LuJw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.04-2022.11.21-000094 KPby4uPXRMePMqOP0pvHyA   3   0          0            0       678b           678b
green  open   .siem-signals-default-000002              rS6GZpKUQ-2AcOf2E0tyXQ   1   1          0            0       452b           226b
green  open   .siem-signals-default-000003              TvDUIlIhSs-Hivahcznqvw   1   1          0            0       452b           226b
green  open   .monitoring-es-7-2022.11.20               2Ac9vd67Q96QlQ68h7ohXw   1   1    2811306       916493      2.1gb            1gb
green  open   .monitoring-es-7-2022.11.21               Mc9e4_NuSAWM9xuHF4Vsbw   1   1    2845512       586684      2.1gb            1gb
green  open   .siem-signals-default-000004              t5UR8ZALSUqB-3VJwHBWWg   1   1          0            0       452b           226b
green  open   .monitoring-es-7-2022.11.22               QidsNqROQleXw2rqAY757g   1   1     594244       305204    823.5mb          412mb
green  open   .siem-signals-default-000005              dEqsBmZoSgqrv_a3iKkTWg   1   1          0            0       452b           226b
green  open   .siem-signals-default-000006              svpNu_GDRwSrZRZoPcI7mA   1   1          0            0       452b           226b
green  open   .ds-logstash-2052.08.12-2022.11.21-000001 34VfAccLTu2NJywHFDTVWQ   3   0          3            0     19.5kb         19.5kb
green  open   .ds-logstash-2022.09.22-2022.11.18-000108 qWyeXRJhRiK1kuTkY9srgw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.20-2022.11.18-000116 tnhKTDl3R8uJw_ZGUXq4gA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.29-2022.11.18-000098 t76C4-jvQ6KUlrJ9RK6xkA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.17-2022.11.21-000006 lIWMcz_iRNCZiPWjItQJEw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.20-2022.11.19-000118 nwRElda0RNaZQ7OGRpyUbw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.15-2022.11.19-000006 HHJP54m5S4-isRyjcHvuoA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.15-2022.11.22-000012 u3acMkiAQnmKE6ILpKjjUw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.21-2022.11.19-000056 _3UcY8wCRTSlT1lwgOAeBQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.17-2022.11.18-000122 FpEwLbvPQcWKPHnEiAN8tw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.01-2022.11.19-000096 VHt95ordRIax71LvCvLCMg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.18-2022.11.19-000062 p2FXvmMtSF-vhcvp1B_QiQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.22-2022.11.21-000114 2zEf7HxoS72DvqNZerlLeQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.14-2022.11.19-000008 xQ0Vk9-fSRSge1_Y8sLE1Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.17-2022.11.19-000124 Q0LuT9dvT3u8bY7zzOXpkQ   3   0          0            0       678b           678b
green  open   .monitoring-es-7-2022.11.16               2UrLZiCfSRqmqhTBbFgq-g   1   1    2672435       885328      1.9gb       1005.9mb
green  open   .monitoring-es-7-2022.11.17               nImR5EwgQ_SH8JkcMeL9NA   1   1    2659080       121403      1.8gb        971.7mb
green  open   .monitoring-es-7-2022.11.18               jJtuAEYIQF2HmSNreLRDkw   1   1    2760676       658335        2gb            1gb
green  open   .ds-logstash-2022.09.22-2022.11.20-000112 1DGgQTNtQYWxSRkG4nUChw   3   0          0            0       678b           678b
green  open   .monitoring-es-7-2022.11.19               g9JwRF1YS1CJWnuOxFzBWg   1   1    2784606       875754      2.1gb            1gb
green  open   .ds-logstash-2022.09.19-2022.11.18-000102 rr_sG_dISpilvteIPR8mNw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.01-2022.11.21-000038 8Za0aBf8TKGxGMs7JqaL4g   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.14-2022.11.21-000012 7BfxZbv5SeOgBI2MeZDv-Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.03-2022.11.18-000090 Iob_-8lSTUKgL_L2o26RhQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.12-2022.11.19-000012 a89G2RrwRCecIvOHM0mMBg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.15-2022.11.20-000008 H-1MxTX-TrGuFRxJ-aGM2w   3   0          0            0       678b           678b
green  open   .fleet-enrollment-api-keys-7              eMNUDegcSRGLQCXHgyl1Eg   1   1          2            0     13.3kb          6.6kb
green  open   .apm-agent-configuration                  RspLMi00R0W4AZ9QGI_Hcw   1   1          1            0     10.2kb          5.1kb
green  open   .ds-logstash-2022.10.22-2022.11.21-000058 wLSA9L4cQQWjpKk-AHilqA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.20-2022.11.21-000122 tuI0mN1UR26SZzZoP01CHA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.20-2022.11.19-000058 EA_J_S6YQ4qH2QgO_cLCOw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.03-2022.11.21-000096 7qLyqLt3SL-CcjrDniCuWA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.21-2022.11.21-000098 0EwtsN3xS7mtAKdX1nNcWA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.19-2022.11.18-000058 G8U6sHIiR4STN5XiZmuPGQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.23-2022.11.21-000056 3yDdLrVfQsivd_KAgW3CzA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.25-2022.11.20-000050 IvGhlK2lQt2O6T00Xq2i-A   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.17-2022.11.19-000064 EtQU3STTRm2xxcwEpB9_UQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.22-2022.11.18-000052 nSR4rttcSL6_uxpUSHeqfA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.07-2022.11.21-000088 dkNw4vOFQHSOSNIbMu5NNw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.16-2022.11.20-000006 Y99wQ8sWReCUUQ_Ya-ep9g   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.11-2022.11.19-000014 c7tpRuWoSvej0QVJZEAAaA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.05-2022.11.20-000090 0nULtzgAQ36oWOxnm8Bi6Q   3   0          0            0       678b           678b
green  open   .kibana_task_manager_7.17.2_001           R7z0qznjScGokIRpKLyf6w   1   1         32       448536    115.1mb         57.6mb
green  open   .ds-logstash-2022.09.21-2022.11.18-000092 msygbRpfSVu-rou1zHmi3Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.18-2022.11.20-000064 mEpf48mrSNKs890jC8y27w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.30-2022.11.22-000044 TficYa9wTsqeyE8OigsW3g   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.10-2022.11.21-000082 SGyeQBykRju4GrUfOt5iwA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.31-2022.11.20-000038 stQ2gUoCTi-DfjkPT4mVDw   3   0          0            0       678b           678b
green  open   .kibana_7.17.2_001                        MOPb_zaOS4yg4Kv39k2Q-w   1   1       3515       168867     73.1mb         36.5mb
green  open   .ds-logstash-2022.09.30-2022.11.20-000098 NYo-EKZsTPWzpcHaHN2C2g   3   0          
green  open   .ds-logstash-2022.11.21-2022.11.22-000002 EHOUrJvqRBGrIbBBcsUfog   3   0       9634            0      5.5mb          5.5mb
green  open   .ds-logstash-2022.10.27-2022.11.22-000050 L62t5fyaQYyBhylA0n9T7Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.21-2022.11.20-000058 jNg83KKCToKfcb__9ZblBg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.04-2022.11.21-000032 YpDudqp8TS2QSCaF4XMgwQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.28-2022.11.18-000094 07aVw0F0Qyedvddid9TZFg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.02-2022.11.19-000032 KxSztestSXi3Bf-yb5lCPg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.13-2022.11.18-000070 0BQCCi7WTZuvlTGv8DVLaw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.20-2022.11.21-000002 dIjHy_K_Q_GS5neLzPF3xw   3   0       8440            0      4.8mb          4.8mb
green  open   .ds-logstash-2022.11.18-2022.11.22-000006 75mfu_rcRgik1Vv0v-7UgA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.16-2022.11.21-000102 W0iRXMCQQLyww_R3qLR7eA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.26-2022.11.21-000050 FJ94BYwUTfimq7bMillzvQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.18-2022.11.20-000104 qeJskNxRSyCOZu98DY7fQA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.30-2022.11.19-000038 8bNUufFMRXaiGliFlhaE2w   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.13-2022.11.21-000076 610h3HBqSFiQlDAqG2TbLw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.26-2022.11.18-000104 zPFnGRyoQwSNOyL3NlB8hg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.16-2022.11.19-000098 a0TMJEQiRq-hNS33sqiPHQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.02-2022.11.20-000096 KHVSGR_HSjad3QzYtU8Smw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.27-2022.11.19-000044 CwUppADKS6y0XEUVYuAQTA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.15-2022.11.20-000070 IglGvcwFQxu3DaoQ--GphA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.01-2022.11.21-000100 Qz6epXwJQi6jBvFM8PjjVQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.06-2022.11.20-000026 c24ZHhUgTG2qajGekXp7EA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.19-2022.11.20-000002 cGPUpGkUTs-wzzcY5VMjJg   3   0       6589            0      2.8mb          2.8mb
green  open   .ds-logstash-2022.10.14-2022.11.20-000072 XziDOfSkRxuoESClu4f_LA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.09-2022.11.19-000080 GHkWQxXaSkeH45264U8dsQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.07-2022.11.21-000026 9TINEeJtTIqQBxl9lSpJSg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.27-2022.11.19-000104 nztKYPbuTW-qlvsMmSw5iw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.05-2022.11.21-000030 G70NxezXSGyhdogv_U9E6w   3   0          0            0       678b           678b
green  open   .apm-custom-link                          7wOJQZLFToCFpdsMkuiyQQ   1   1          0            0       452b           226b
green  open   .ds-logstash-2022.10.16-2022.11.20-000068 HrXWK9RNQmuzd03eRlT_mA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.11-2022.11.19-000076 jjOgm812QTy73VfOWHR6Bg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.25-2022.11.21-000096 yFQAOOSITB2sTa3_Br2FHQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.08-2022.11.19-000082 LshHHXhCRoKqfxnLSkL3EA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.10-2022.11.21-000020 B72Ya9PaQziOO96lwmqrHg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.25-2022.11.18-000090 ADECGqlMSiiPWjs3HHOpxA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.13-2022.11.19-000072 AR4vISlqRsKbXn2jBx_dXA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.18-2022.11.21-000106 QJ71tnzPRUqzQWSg2UWQww   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.13-2022.11.20-000074 PWHy8kFWRzC4-CSWcaQ2Cg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.09-2022.11.20-000082 TSQ7QFnPQE-85j2giH-MtA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.26-2022.11.19-000106 74McnuYgQxe_O8fCdCHJEw   3   0          0            0       678b           678b
green  open   .tasks                                    wMHeCndLR5mg3U9ouEHGdA   1   1        230            0    446.6kb        217.2kb
green  open   .ds-logstash-2022.09.19-2022.11.21-000108 vBrIdz29Txyhm1vDRR_OVg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.12-2022.11.20-000076 iBVWmFtST6qHTUAV5h7y9A   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.31-2022.11.21-000040 iX2caldKQtenNIFWS3WBsg   3   0          0            0       678b           678b
green  open   .fleet-policies-7                         WmA_Mz9_SluWWp-3ZjDopA   1   1          3            0     28.6kb         14.3kb
green  open   .ds-logstash-2022.10.12-2022.11.19-000074 R7HkZXQSR9esGQaPDHLEbw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.17-2022.11.20-000066 XqOrFX45TySU2RR3CwTHgg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.06-2022.11.21-000028 V9A7IS5aTKGeplSjfRSnWg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.30-2022.11.21-000042 bgy022TmR1OfGjjVPkTBvw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.08-2022.11.20-000084 FpU4h691Qq2H4bystr8R2Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.22-2022.11.22-000001 xrvNKn7cQqCGsnOMRDVyMQ   3   0    9411784            0      5.8gb          5.8gb
green  open   .ds-logstash-2022.10.28-2022.11.21-000046 81iSNDcNSyOeE8H-TQKnYA   3   0          0            0       678b           678b
green  open   .geoip_databases                          xQ7KyV9xTk6OmVmp5R0-FA   1   1         41           41     81.3mb         39.3mb
green  open   .ds-logstash-2022.10.04-2022.11.19-000090 Y1pwzd8GRP-qLdZ8S3kLiQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.05-2022.11.19-000088 xfSBNE8JQa2Il1sfNst3Fg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.19-2022.11.20-000062 P1fWmbCkRj-4gPGXKPlrHQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.15-2022.11.19-000068 Ly6sHquaSWeD-Tfm_i32Bw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.03-2022.11.21-000034 JrXgXGhlTeWWD72wpzBepQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.17-2022.11.19-000003 pAmgYTLBQZGOEWR0izB4MA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.13-2022.11.21-000014 exme0ZpjS22YnC8I8q-zCg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.11-2022.11.20-000078 B66QeUXWRuyTEIRTJX4IkA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.27-2022.11.20-000106 b3BEF-I6R0e0SJhzvJTMwg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.17-2022.11.21-000128 FFl-uKmXSRir-J-N8XQXwQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.11-2022.11.21-000018 VyzpXxg2QNG-OGVtzgfzZw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.19-2022.11.19-000001 aRAGSvsISIS1iQRepTE1hQ   3   0   49124078            0     20.1gb         20.1gb
green  open   .ds-logstash-2022.11.18-2022.11.20-000003 L9NImBU7R2yt1ehazhQ4nQ   3   0         57            0    489.5kb        489.5kb
green  open   .ds-logstash-2022.10.22-2022.11.20-000056 oLWtzwagS5Syd1zBHQSEOA   3   0          0            0       678b           678b
green  open   .ds-waleed-logs-2022.11.22-000155         nblba5_FQH-FZNq7miGV9Q   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.06-2022.11.20-000088 aTNQWORxRF2lNTPHMfQePg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.17-2022.11.22-000008 vjtBqJvhQ2uzPgTWUcxUYw   3   0          0            0       678b           678b
green  open   .ds-waleed-logs-2022.11.21-000145         bijOKbxESoeU7MdXrIF7vQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.13-2022.11.19-000010 EhpVsXEGS_OEIXsRwxcmEw   3   0          0            0       678b           678b
green  open   .ds-waleed-logs-2022.11.21-000143         8X22UXEMSMSl_EDaYTQ6cw   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.18           lnciK_MISdqqCCkVR1KHBg   1   1      17256            0     14.4mb          8.7mb
green  open   .ds-waleed-logs-2022.11.21-000141         PrWh1smOStm6mp5R4Hlaeg   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.17           OmM_iWYoQFC2el6-bRlc6w   1   1      17186            0      7.9mb          3.9mb
green  open   .ds-logstash-2022.11.15-2022.11.21-000010 lnvlch12Rh6UimxjwWAScQ   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.19           BzHoQZeeR7quA0O1PktAJg   1   1      17280            0     13.9mb          8.8mb
green  open   .ds-logstash-2022.10.21-2022.11.18-000054 e7JnrD5aRa6GLQ63WqW3uQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.01-2022.11.18-000094 SjWHgjJiSQq-YLxC4-1VNQ   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.21           ENzTX9zZSnazKjra71IEXA   1   1      17278            0     16.3mb          7.9mb
green  open   .ds-logstash-2022.10.26-2022.11.20-000048 2y4ILc52SqWR7W_Fec0B6A   3   0          0            0       678b           678b
green  open   .ds-waleed-logs-2022.11.21-000149         X_AaUPyWSdurb29KjfYrnA   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.20           OT7ypwzhRTidPn0PdyybuA   1   1      17280            0     14.7mb          7.1mb
green  open   .ds-waleed-logs-2022.11.21-000147         9yPrBDyHRNuqYvMVU-YlPw   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.22           kFQ1yKw7Sk6WEUK0Y7f7jg   1   1       3344            0      4.2mb          1.5mb
green  open   .ds-logstash-2022.11.16-2022.11.22-000010 JB_sgbX1SvS5s1lcch_vHA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.22-2022.11.19-000110 7JDEkzFyR_O_X35YQzUjqQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.18-2022.11.18-000060 s26UtilhRKCkx6AVjZZ-Bg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.16-2022.11.19-000004 0UybH18YSUm7suME7UIJpg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.20-2022.11.22-000003 T1gyU7R1RIi_Is5DUgp9bg   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.11.18-2022.11.21-000004 Djb2ViczTKu9zLGgy0-wZg   3   0        238            0      563kb          563kb
green  open   .ds-logstash-2022.11.20-2022.11.20-000001 Juh5MMlZTjShBQp_DYcNvg   3   0   42881250            0     19.1gb         19.1gb
green  open   .ds-waleed-logs-2022.11.21-000153         doR6XZR7Ts67-OnNmY7YmQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.10.26-2022.11.22-000052 KTkeP7cFQguKNsUdBfxpww   3   0          0            0       678b           678b
green  open   .ds-waleed-logs-2022.11.21-000151         J32oMx_jSNW_Tph27kAUWA   3   0          0            0       678b           678b
green  open   .monitoring-kibana-7-2022.11.16           Ad-Mw0XPTqOirMOxAy6sAg   1   1      17202            0      7.7mb          3.9mb
green  open   .ds-logstash-2022.10.25-2022.11.22-000054 kuXtH5I4SqSJnZ7auad9YQ   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.16-2022.11.18-000096 T44P_wCZRTKPAmfRLwQJkw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.21-2022.11.19-000094 FwpdvRPqR06DJXPTdy-Qdw   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.23-2022.11.21-000096 hpGcq9BbQpaYgrdfpaCVTA   3   0          0            0       678b           678b
green  open   .ds-logstash-2022.09.23-2022.11.18-000090 ViDtEkRNSLyHiOSkGipRjA   3   0          0            0       678b           678b

Here is an explain of an index I would have expected to be deleted already since the date range is from 23 Sep.
GET .ds-logstash-2022.09.23-2022.11.18-000090/_ilm/explain

  "indices" : {
    ".ds-logstash-2022.09.23-2022.11.18-000090" : {
      "index" : ".ds-logstash-2022.09.23-2022.11.18-000090",
      "managed" : true,
      "policy" : "logstash-lifecycle-policy",
      "lifecycle_date_millis" : 1668868934988,
      "age" : "2.58d",
      "phase" : "hot",
      "phase_time_millis" : 1668781935613,
      "action" : "complete",
      "action_time_millis" : 1668868935988,
      "step" : "complete",
      "step_time_millis" : 1668868935988,
      "phase_execution" : {
        "policy" : "logstash-lifecycle-policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "48gb",
              "max_age" : "1d"
            }
          }
        },
        "version" : 6,
        "modified_date_in_millis" : 1668501109372
      }
    }
  }
}

Here is the request for the ILM itself

PUT _ilm/policy/logstash-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "48gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Well I am not sure where the index names are coming from but right there it says there index is only 2.58 days old and your policy says delete 3days after rollover so that is most likely why it has not been deleted yet.

Also something else going on those are empty indices.. ILM does not delete docs it deletes indices.

Perhaps share your logstash pipeline output section.

Thanks. Yes agreed that is the reason why it's not deleted. The strange part for me is why is it still being written to up until the the 18th Nov when the index looks to be created on the 23 Sep already

It's using fluent-bit to push the logs. Below is the config which I think is relevent.

luaScripts: {}

## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file
config:
  service: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser cri
        Parser cri
        Tag kube.*
        Mem_Buf_Limit 32MB
        Rotate_Wait 10
        Skip_Long_Lines On
        DB /var/log/flb_kube.db
        #Refresh_Interval 10

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Merge_Log_Trim      On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Keep_Log            Off

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Host elasticsearch-es-http
        Port 9200
        HTTP_User elastic
        HTTP_Passwd -----------
        tls On
        tls.verify Off
        Logstash_Prefix logstash
        Logstash_Format On
        Replace_Dots On
        Retry_Limit False

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        multi_line
        Format      regex
        Regex       (?<log>^{"log":"\d{4}-\d{2}-\d{2}.*)

    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

# The config volume is mounted by default, either to the existingConfigMap value, or the default of "fluent-bit.fullname"
volumeMounts:
  - name: config
    mountPath: /fluent-bit/etc/fluent-bit.conf
    subPath: fluent-bit.conf
  - name: config
    mountPath: /fluent-bit/etc/custom_parsers.conf
    subPath: custom_parsers.conf

It looks like the issue was with my fluent-bit configuration. I added the following options that have an asterisk, which remove the date range from the indexes and the ILM kicked in correctly

  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Host elasticsearch-es-http
        Port 9200
        HTTP_User -------
        HTTP_Passwd ------
        tls On
        tls.verify Off
        **Logstash_Prefix fluent-bit**
**        Logstash_Format Off**
**        Index fluent-bit**
        Replace_Dots On
        Retry_Limit False
1 Like