Hi There,
I am making use of Data Streams and an ILM Policy to delete data that is rolled over from Hot phase to delete phase, this happens after 3 days.
I wanted to check if there is a way to include the actual deletion of the Logstash Data Streams themselves as part of the ILM. The ILM deletes the data in the indexes but the data stream backing indexes still seem to exist, as well as the Data Streams themselves.
This pushes the index and shard count up over time.
Is there a correct way to maintain the data streams themselves and to clear out the backing indexes automatically as part of an ILM or another method?
Thanks
ILM deletes things at the index level. If you are still seeing backing indices then we'd need more information to help. Can you share the output from _cat/indices?v and an ILM explain on the policy?
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .ds-logstash-2022.10.29-2022.11.19-000040 NAZ463lgRUKrBlSjljjgIQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.14-2022.11.21-000074 0tJbahISTgarbQ4ZvOUX3A 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.06-2022.11.22-000030 Q_kNNw7rTuCs6EgWR-3lQw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.03-2022.11.19-000030 ajgbj8ecQiWqK6L1Tgylbw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.07-2022.11.22-000028 4OKJ5yygTmmg5cPcv77N_A 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.05-2022.11.19-000026 BWPPLcDbSse2-HLR5v0NvA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.05-2022.11.22-000032 j_CdEO8rR9-cb9OZ6-BatA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.27-2022.11.18-000102 mxRhZP4qSjKivQJmnOJJ2w 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.11-2022.11.18-000074 3laZdYA2TnSSB7AxWiFCnw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.31-2022.11.19-000036 EFi6A05hQe2g3fRaAt2E7Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.25-2022.11.19-000092 I5dHLWYkQ767EseqpjU6BA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.28-2022.11.19-000042 fpNx5qjJRtqVoeUB1Ln0Rw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.07-2022.11.20-000024 F8hNC3ilQVmWS9hd3gCm1Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.04-2022.11.19-000028 Us1-4GJARgCKDe7hFzawHQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.25-2022.11.20-000094 q30w-8nER8G4eZQ3xVvnJg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.10-2022.11.20-000018 RZ7uGpV-Tr-O-YKt4ZON7Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.08-2022.11.18-000080 iWGY2IHpT6KMZww0ltqPtg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.11-2022.11.20-000016 9m_SefCXTMyy83yjkwr52Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.04-2022.11.22-000034 NrpkOBHAQtKq-v-oQ0gIsg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.08-2022.11.22-000026 qIH_JqsaSgq_1lzrBauHGA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.16-2022.11.20-000100 bMEFy1aISK6rjwTpIHQO3w 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.11-2022.11.21-000080 iX1yA6dxTG2uPB2hoUnnhg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.07-2022.11.18-000082 ID50efu1TMW3gwFwRTZI7w 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.10-2022.11.18-000076 U7W8FPsqSlSsXKMEpLWzQQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.08-2022.11.20-000022 T8gBt-uAQ0Szn_nvz8bnjw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.16-2022.11.21-000070 4Q4Vai6bR_yxB3Ykok1o2g 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.28-2022.11.21-000100 jdvKs1xuScOHqE3wlZ4HGA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.09-2022.11.20-000020 Y9Y95zG2TZeJT8wc-kcEVg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.09-2022.11.22-000024 fiw_JkRdSl-Kdl7bjIrO3Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.12-2022.11.22-000018 8zvFbPkwSn22e6Z5drQPqQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.29-2022.11.21-000104 -imM346WQPO1Y8TWipb1qA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.15-2022.11.21-000072 vgu9pUNeRGecI2qcB8--kw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.23-2022.11.19-000092 AzbovF2oRku1GVvVHg5-cQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.14-2022.11.18-000068 4G20sHYdS0mWNMWYCvWVKQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.04-2022.11.20-000030 9oNJEsK6RDWbme1mNFatFw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.26-2022.11.19-000046 A6XDQAnhSTu-ehXGux0I4g 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.27-2022.11.21-000108 ylLYvbqJR0q6ajS7s-AEmg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.01-2022.11.20-000098 Boj0n7ScTR-FAJzdf2FNmw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.23-2022.11.20-000094 PHk_8s6cRJyoymlJlN0-5g 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.08-2022.11.19-000020 jxdpqwqATRWn9PYJYF43DQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.02-2022.11.20-000034 BaCWQ_0KTAyYjL7zMFB0kA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.06-2022.11.18-000084 33fE5Te-QFmyG2OJdFckXg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.28-2022.11.22-000048 nd_LUyiZRzaJpli7OSXgtw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.20-2022.11.21-000062 80AOzBtfQFuAxPUX0O-eOg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.30-2022.11.20-000040 FY7OAAbBS9CTq4pq3hASKw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.28-2022.11.20-000098 OYprzBpcSSC4iseSEb9wgw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.19-2022.11.21-000064 7uKP1xAiR5CuBH8blrfL7Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.15-2022.11.18-000066 WfDL4fIhQLu9nkRlWT2yIA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.29-2022.11.20-000042 YqQ_U_OuTiCE-F9xSrI5sA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.13-2022.11.20-000012 dXCPzqwaSY21sXCgVsZJEw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.03-2022.11.22-000036 hI3PQ2xGTGmR-UeG3u7GiA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.01-2022.11.22-000040 4-ACXKuHR9iRCDji39pAZA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.21-2022.11.21-000001 BC55po1oT8KYlmmiVPy8cw 3 0 35710914 0 15.7gb 15.7gb
green open .ds-logstash-2022.11.09-2022.11.19-000018 g1Zp4DIhSQOVonPoT5oVgQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.11-2022.11.22-000020 geVg1jhUSkKA-H8Lqsdm4w 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.05-2022.11.20-000028 0UWEZ_FcS1Wg_KVh30-uog 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.04-2022.11.18-000088 qZHJJZnAQQStHiUBBARtJA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.14-2022.11.22-000014 UiSNWGmxTX2Zjog05Y2Uew 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.06-2022.11.19-000024 zmayhGntSPqTIj7IQfX33Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.30-2022.11.21-000100 9GXaX8UjR8SLWnjRsgDTGw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.23-2022.11.19-000052 tp2TyEicSpi4UsqhNJ_PCA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.12-2022.11.21-000078 8HYhJHCYSkyraxR1lNNR9w 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.06-2022.11.21-000090 yr-ZC0H7Te28IvWRyGXzag 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.20-2022.11.18-000056 ksnJsAy7QWWF93V1inVGIA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.09-2022.11.21-000084 q7W7kQElT6-x8sYypyzksQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.03-2022.11.20-000094 eP8tVUJHRrWFeeuV88V4XQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.21-2022.11.20-000096 HyWbLvlKQ3Csisty8h4exg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.01-2022.11.19-000034 AZkTrpUCQCCbIz5VXZuZlA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.09-2022.11.18-000078 oZd-BsPBTZu6iBBpZ0JhOw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.19-2022.11.20-000106 dUUqJMpBSU-hOrN2v_4fGA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.17-2022.11.18-000062 gQ0bjwPrTpSxjTOCuz7fmw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.31-2022.11.22-000042 KCI7o2vsSGCFQFzns0h_Ug 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.30-2022.11.19-000096 -IeNVkYyRXi92VedwBXMnQ 3 0 0 0 678b 678b
green open .async-search C3YFN9rISJWJW4U138Nxow 1 1 0 0 528b 264b
green open .ds-logstash-2022.09.18-2022.11.19-000102 cUpnmn-iR7qdoHw17DN0bQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.27-2022.11.20-000046 VaJAFflNStilQGHMdO8mtg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.17-2022.11.21-000068 8vC1GTXwQLqY0mkcHiwjwQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.12-2022.11.18-000072 wN5VK1VHQRSHNRw5matLSA 3 0 0 0 678b 678b
green open .siem-signals-default-000001 BaHhEx-jSLCB1G_vft_iug 1 1 0 0 452b 226b
green open .ds-logstash-2022.10.24-2022.11.20-000052 cVBueA5ETjeEKul029LuJw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.04-2022.11.21-000094 KPby4uPXRMePMqOP0pvHyA 3 0 0 0 678b 678b
green open .siem-signals-default-000002 rS6GZpKUQ-2AcOf2E0tyXQ 1 1 0 0 452b 226b
green open .siem-signals-default-000003 TvDUIlIhSs-Hivahcznqvw 1 1 0 0 452b 226b
green open .monitoring-es-7-2022.11.20 2Ac9vd67Q96QlQ68h7ohXw 1 1 2811306 916493 2.1gb 1gb
green open .monitoring-es-7-2022.11.21 Mc9e4_NuSAWM9xuHF4Vsbw 1 1 2845512 586684 2.1gb 1gb
green open .siem-signals-default-000004 t5UR8ZALSUqB-3VJwHBWWg 1 1 0 0 452b 226b
green open .monitoring-es-7-2022.11.22 QidsNqROQleXw2rqAY757g 1 1 594244 305204 823.5mb 412mb
green open .siem-signals-default-000005 dEqsBmZoSgqrv_a3iKkTWg 1 1 0 0 452b 226b
green open .siem-signals-default-000006 svpNu_GDRwSrZRZoPcI7mA 1 1 0 0 452b 226b
green open .ds-logstash-2052.08.12-2022.11.21-000001 34VfAccLTu2NJywHFDTVWQ 3 0 3 0 19.5kb 19.5kb
green open .ds-logstash-2022.09.22-2022.11.18-000108 qWyeXRJhRiK1kuTkY9srgw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.20-2022.11.18-000116 tnhKTDl3R8uJw_ZGUXq4gA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.29-2022.11.18-000098 t76C4-jvQ6KUlrJ9RK6xkA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.17-2022.11.21-000006 lIWMcz_iRNCZiPWjItQJEw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.20-2022.11.19-000118 nwRElda0RNaZQ7OGRpyUbw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.15-2022.11.19-000006 HHJP54m5S4-isRyjcHvuoA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.15-2022.11.22-000012 u3acMkiAQnmKE6ILpKjjUw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.21-2022.11.19-000056 _3UcY8wCRTSlT1lwgOAeBQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.17-2022.11.18-000122 FpEwLbvPQcWKPHnEiAN8tw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.01-2022.11.19-000096 VHt95ordRIax71LvCvLCMg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.18-2022.11.19-000062 p2FXvmMtSF-vhcvp1B_QiQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.22-2022.11.21-000114 2zEf7HxoS72DvqNZerlLeQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.14-2022.11.19-000008 xQ0Vk9-fSRSge1_Y8sLE1Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.17-2022.11.19-000124 Q0LuT9dvT3u8bY7zzOXpkQ 3 0 0 0 678b 678b
green open .monitoring-es-7-2022.11.16 2UrLZiCfSRqmqhTBbFgq-g 1 1 2672435 885328 1.9gb 1005.9mb
green open .monitoring-es-7-2022.11.17 nImR5EwgQ_SH8JkcMeL9NA 1 1 2659080 121403 1.8gb 971.7mb
green open .monitoring-es-7-2022.11.18 jJtuAEYIQF2HmSNreLRDkw 1 1 2760676 658335 2gb 1gb
green open .ds-logstash-2022.09.22-2022.11.20-000112 1DGgQTNtQYWxSRkG4nUChw 3 0 0 0 678b 678b
green open .monitoring-es-7-2022.11.19 g9JwRF1YS1CJWnuOxFzBWg 1 1 2784606 875754 2.1gb 1gb
green open .ds-logstash-2022.09.19-2022.11.18-000102 rr_sG_dISpilvteIPR8mNw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.01-2022.11.21-000038 8Za0aBf8TKGxGMs7JqaL4g 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.14-2022.11.21-000012 7BfxZbv5SeOgBI2MeZDv-Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.03-2022.11.18-000090 Iob_-8lSTUKgL_L2o26RhQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.12-2022.11.19-000012 a89G2RrwRCecIvOHM0mMBg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.15-2022.11.20-000008 H-1MxTX-TrGuFRxJ-aGM2w 3 0 0 0 678b 678b
green open .fleet-enrollment-api-keys-7 eMNUDegcSRGLQCXHgyl1Eg 1 1 2 0 13.3kb 6.6kb
green open .apm-agent-configuration RspLMi00R0W4AZ9QGI_Hcw 1 1 1 0 10.2kb 5.1kb
green open .ds-logstash-2022.10.22-2022.11.21-000058 wLSA9L4cQQWjpKk-AHilqA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.20-2022.11.21-000122 tuI0mN1UR26SZzZoP01CHA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.20-2022.11.19-000058 EA_J_S6YQ4qH2QgO_cLCOw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.03-2022.11.21-000096 7qLyqLt3SL-CcjrDniCuWA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.21-2022.11.21-000098 0EwtsN3xS7mtAKdX1nNcWA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.19-2022.11.18-000058 G8U6sHIiR4STN5XiZmuPGQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.23-2022.11.21-000056 3yDdLrVfQsivd_KAgW3CzA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.25-2022.11.20-000050 IvGhlK2lQt2O6T00Xq2i-A 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.17-2022.11.19-000064 EtQU3STTRm2xxcwEpB9_UQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.22-2022.11.18-000052 nSR4rttcSL6_uxpUSHeqfA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.07-2022.11.21-000088 dkNw4vOFQHSOSNIbMu5NNw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.16-2022.11.20-000006 Y99wQ8sWReCUUQ_Ya-ep9g 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.11-2022.11.19-000014 c7tpRuWoSvej0QVJZEAAaA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.05-2022.11.20-000090 0nULtzgAQ36oWOxnm8Bi6Q 3 0 0 0 678b 678b
green open .kibana_task_manager_7.17.2_001 R7z0qznjScGokIRpKLyf6w 1 1 32 448536 115.1mb 57.6mb
green open .ds-logstash-2022.09.21-2022.11.18-000092 msygbRpfSVu-rou1zHmi3Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.18-2022.11.20-000064 mEpf48mrSNKs890jC8y27w 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.30-2022.11.22-000044 TficYa9wTsqeyE8OigsW3g 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.10-2022.11.21-000082 SGyeQBykRju4GrUfOt5iwA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.31-2022.11.20-000038 stQ2gUoCTi-DfjkPT4mVDw 3 0 0 0 678b 678b
green open .kibana_7.17.2_001 MOPb_zaOS4yg4Kv39k2Q-w 1 1 3515 168867 73.1mb 36.5mb
green open .ds-logstash-2022.09.30-2022.11.20-000098 NYo-EKZsTPWzpcHaHN2C2g 3 0
green open .ds-logstash-2022.11.21-2022.11.22-000002 EHOUrJvqRBGrIbBBcsUfog 3 0 9634 0 5.5mb 5.5mb
green open .ds-logstash-2022.10.27-2022.11.22-000050 L62t5fyaQYyBhylA0n9T7Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.21-2022.11.20-000058 jNg83KKCToKfcb__9ZblBg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.04-2022.11.21-000032 YpDudqp8TS2QSCaF4XMgwQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.28-2022.11.18-000094 07aVw0F0Qyedvddid9TZFg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.02-2022.11.19-000032 KxSztestSXi3Bf-yb5lCPg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.13-2022.11.18-000070 0BQCCi7WTZuvlTGv8DVLaw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.20-2022.11.21-000002 dIjHy_K_Q_GS5neLzPF3xw 3 0 8440 0 4.8mb 4.8mb
green open .ds-logstash-2022.11.18-2022.11.22-000006 75mfu_rcRgik1Vv0v-7UgA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.16-2022.11.21-000102 W0iRXMCQQLyww_R3qLR7eA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.26-2022.11.21-000050 FJ94BYwUTfimq7bMillzvQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.18-2022.11.20-000104 qeJskNxRSyCOZu98DY7fQA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.30-2022.11.19-000038 8bNUufFMRXaiGliFlhaE2w 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.13-2022.11.21-000076 610h3HBqSFiQlDAqG2TbLw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.26-2022.11.18-000104 zPFnGRyoQwSNOyL3NlB8hg 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.16-2022.11.19-000098 a0TMJEQiRq-hNS33sqiPHQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.02-2022.11.20-000096 KHVSGR_HSjad3QzYtU8Smw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.27-2022.11.19-000044 CwUppADKS6y0XEUVYuAQTA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.15-2022.11.20-000070 IglGvcwFQxu3DaoQ--GphA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.01-2022.11.21-000100 Qz6epXwJQi6jBvFM8PjjVQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.06-2022.11.20-000026 c24ZHhUgTG2qajGekXp7EA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.19-2022.11.20-000002 cGPUpGkUTs-wzzcY5VMjJg 3 0 6589 0 2.8mb 2.8mb
green open .ds-logstash-2022.10.14-2022.11.20-000072 XziDOfSkRxuoESClu4f_LA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.09-2022.11.19-000080 GHkWQxXaSkeH45264U8dsQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.07-2022.11.21-000026 9TINEeJtTIqQBxl9lSpJSg 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.27-2022.11.19-000104 nztKYPbuTW-qlvsMmSw5iw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.05-2022.11.21-000030 G70NxezXSGyhdogv_U9E6w 3 0 0 0 678b 678b
green open .apm-custom-link 7wOJQZLFToCFpdsMkuiyQQ 1 1 0 0 452b 226b
green open .ds-logstash-2022.10.16-2022.11.20-000068 HrXWK9RNQmuzd03eRlT_mA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.11-2022.11.19-000076 jjOgm812QTy73VfOWHR6Bg 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.25-2022.11.21-000096 yFQAOOSITB2sTa3_Br2FHQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.08-2022.11.19-000082 LshHHXhCRoKqfxnLSkL3EA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.10-2022.11.21-000020 B72Ya9PaQziOO96lwmqrHg 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.25-2022.11.18-000090 ADECGqlMSiiPWjs3HHOpxA 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.13-2022.11.19-000072 AR4vISlqRsKbXn2jBx_dXA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.18-2022.11.21-000106 QJ71tnzPRUqzQWSg2UWQww 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.13-2022.11.20-000074 PWHy8kFWRzC4-CSWcaQ2Cg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.09-2022.11.20-000082 TSQ7QFnPQE-85j2giH-MtA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.26-2022.11.19-000106 74McnuYgQxe_O8fCdCHJEw 3 0 0 0 678b 678b
green open .tasks wMHeCndLR5mg3U9ouEHGdA 1 1 230 0 446.6kb 217.2kb
green open .ds-logstash-2022.09.19-2022.11.21-000108 vBrIdz29Txyhm1vDRR_OVg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.12-2022.11.20-000076 iBVWmFtST6qHTUAV5h7y9A 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.31-2022.11.21-000040 iX2caldKQtenNIFWS3WBsg 3 0 0 0 678b 678b
green open .fleet-policies-7 WmA_Mz9_SluWWp-3ZjDopA 1 1 3 0 28.6kb 14.3kb
green open .ds-logstash-2022.10.12-2022.11.19-000074 R7HkZXQSR9esGQaPDHLEbw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.17-2022.11.20-000066 XqOrFX45TySU2RR3CwTHgg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.06-2022.11.21-000028 V9A7IS5aTKGeplSjfRSnWg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.30-2022.11.21-000042 bgy022TmR1OfGjjVPkTBvw 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.08-2022.11.20-000084 FpU4h691Qq2H4bystr8R2Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.22-2022.11.22-000001 xrvNKn7cQqCGsnOMRDVyMQ 3 0 9411784 0 5.8gb 5.8gb
green open .ds-logstash-2022.10.28-2022.11.21-000046 81iSNDcNSyOeE8H-TQKnYA 3 0 0 0 678b 678b
green open .geoip_databases xQ7KyV9xTk6OmVmp5R0-FA 1 1 41 41 81.3mb 39.3mb
green open .ds-logstash-2022.10.04-2022.11.19-000090 Y1pwzd8GRP-qLdZ8S3kLiQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.05-2022.11.19-000088 xfSBNE8JQa2Il1sfNst3Fg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.19-2022.11.20-000062 P1fWmbCkRj-4gPGXKPlrHQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.15-2022.11.19-000068 Ly6sHquaSWeD-Tfm_i32Bw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.03-2022.11.21-000034 JrXgXGhlTeWWD72wpzBepQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.17-2022.11.19-000003 pAmgYTLBQZGOEWR0izB4MA 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.13-2022.11.21-000014 exme0ZpjS22YnC8I8q-zCg 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.11-2022.11.20-000078 B66QeUXWRuyTEIRTJX4IkA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.27-2022.11.20-000106 b3BEF-I6R0e0SJhzvJTMwg 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.17-2022.11.21-000128 FFl-uKmXSRir-J-N8XQXwQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.11-2022.11.21-000018 VyzpXxg2QNG-OGVtzgfzZw 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.19-2022.11.19-000001 aRAGSvsISIS1iQRepTE1hQ 3 0 49124078 0 20.1gb 20.1gb
green open .ds-logstash-2022.11.18-2022.11.20-000003 L9NImBU7R2yt1ehazhQ4nQ 3 0 57 0 489.5kb 489.5kb
green open .ds-logstash-2022.10.22-2022.11.20-000056 oLWtzwagS5Syd1zBHQSEOA 3 0 0 0 678b 678b
green open .ds-waleed-logs-2022.11.22-000155 nblba5_FQH-FZNq7miGV9Q 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.06-2022.11.20-000088 aTNQWORxRF2lNTPHMfQePg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.17-2022.11.22-000008 vjtBqJvhQ2uzPgTWUcxUYw 3 0 0 0 678b 678b
green open .ds-waleed-logs-2022.11.21-000145 bijOKbxESoeU7MdXrIF7vQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.13-2022.11.19-000010 EhpVsXEGS_OEIXsRwxcmEw 3 0 0 0 678b 678b
green open .ds-waleed-logs-2022.11.21-000143 8X22UXEMSMSl_EDaYTQ6cw 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.18 lnciK_MISdqqCCkVR1KHBg 1 1 17256 0 14.4mb 8.7mb
green open .ds-waleed-logs-2022.11.21-000141 PrWh1smOStm6mp5R4Hlaeg 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.17 OmM_iWYoQFC2el6-bRlc6w 1 1 17186 0 7.9mb 3.9mb
green open .ds-logstash-2022.11.15-2022.11.21-000010 lnvlch12Rh6UimxjwWAScQ 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.19 BzHoQZeeR7quA0O1PktAJg 1 1 17280 0 13.9mb 8.8mb
green open .ds-logstash-2022.10.21-2022.11.18-000054 e7JnrD5aRa6GLQ63WqW3uQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.01-2022.11.18-000094 SjWHgjJiSQq-YLxC4-1VNQ 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.21 ENzTX9zZSnazKjra71IEXA 1 1 17278 0 16.3mb 7.9mb
green open .ds-logstash-2022.10.26-2022.11.20-000048 2y4ILc52SqWR7W_Fec0B6A 3 0 0 0 678b 678b
green open .ds-waleed-logs-2022.11.21-000149 X_AaUPyWSdurb29KjfYrnA 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.20 OT7ypwzhRTidPn0PdyybuA 1 1 17280 0 14.7mb 7.1mb
green open .ds-waleed-logs-2022.11.21-000147 9yPrBDyHRNuqYvMVU-YlPw 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.22 kFQ1yKw7Sk6WEUK0Y7f7jg 1 1 3344 0 4.2mb 1.5mb
green open .ds-logstash-2022.11.16-2022.11.22-000010 JB_sgbX1SvS5s1lcch_vHA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.22-2022.11.19-000110 7JDEkzFyR_O_X35YQzUjqQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.18-2022.11.18-000060 s26UtilhRKCkx6AVjZZ-Bg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.16-2022.11.19-000004 0UybH18YSUm7suME7UIJpg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.20-2022.11.22-000003 T1gyU7R1RIi_Is5DUgp9bg 3 0 0 0 678b 678b
green open .ds-logstash-2022.11.18-2022.11.21-000004 Djb2ViczTKu9zLGgy0-wZg 3 0 238 0 563kb 563kb
green open .ds-logstash-2022.11.20-2022.11.20-000001 Juh5MMlZTjShBQp_DYcNvg 3 0 42881250 0 19.1gb 19.1gb
green open .ds-waleed-logs-2022.11.21-000153 doR6XZR7Ts67-OnNmY7YmQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.10.26-2022.11.22-000052 KTkeP7cFQguKNsUdBfxpww 3 0 0 0 678b 678b
green open .ds-waleed-logs-2022.11.21-000151 J32oMx_jSNW_Tph27kAUWA 3 0 0 0 678b 678b
green open .monitoring-kibana-7-2022.11.16 Ad-Mw0XPTqOirMOxAy6sAg 1 1 17202 0 7.7mb 3.9mb
green open .ds-logstash-2022.10.25-2022.11.22-000054 kuXtH5I4SqSJnZ7auad9YQ 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.16-2022.11.18-000096 T44P_wCZRTKPAmfRLwQJkw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.21-2022.11.19-000094 FwpdvRPqR06DJXPTdy-Qdw 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.23-2022.11.21-000096 hpGcq9BbQpaYgrdfpaCVTA 3 0 0 0 678b 678b
green open .ds-logstash-2022.09.23-2022.11.18-000090 ViDtEkRNSLyHiOSkGipRjA 3 0 0 0 678b 678b
Here is an explain of an index I would have expected to be deleted already since the date range is from 23 Sep.
GET .ds-logstash-2022.09.23-2022.11.18-000090/_ilm/explain
Well I am not sure where the index names are coming from but right there it says there index is only 2.58 days old and your policy says delete 3days after rollover so that is most likely why it has not been deleted yet.
Also something else going on those are empty indices.. ILM does not delete docs it deletes indices.
Perhaps share your logstash pipeline output section.
Thanks. Yes agreed that is the reason why it's not deleted. The strange part for me is why is it still being written to up until the the 18th Nov when the index looks to be created on the 23 Sep already
It's using fluent-bit to push the logs. Below is the config which I think is relevent.
luaScripts: {}
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file
config:
service: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser cri
Parser cri
Tag kube.*
Mem_Buf_Limit 32MB
Rotate_Wait 10
Skip_Long_Lines On
DB /var/log/flb_kube.db
#Refresh_Interval 10
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Merge_Log_Trim On
K8S-Logging.Parser On
K8S-Logging.Exclude On
Keep_Log Off
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-es-http
Port 9200
HTTP_User elastic
HTTP_Passwd -----------
tls On
tls.verify Off
Logstash_Prefix logstash
Logstash_Format On
Replace_Dots On
Retry_Limit False
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name multi_line
Format regex
Regex (?<log>^{"log":"\d{4}-\d{2}-\d{2}.*)
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
# The config volume is mounted by default, either to the existingConfigMap value, or the default of "fluent-bit.fullname"
volumeMounts:
- name: config
mountPath: /fluent-bit/etc/fluent-bit.conf
subPath: fluent-bit.conf
- name: config
mountPath: /fluent-bit/etc/custom_parsers.conf
subPath: custom_parsers.conf
It looks like the issue was with my fluent-bit configuration. I added the following options that have an asterisk, which remove the date range from the indexes and the ILM kicked in correctly
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-es-http
Port 9200
HTTP_User -------
HTTP_Passwd ------
tls On
tls.verify Off
**Logstash_Prefix fluent-bit**
** Logstash_Format Off**
** Index fluent-bit**
Replace_Dots On
Retry_Limit False
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.