# ILM: Need help organizing my indexes

**URL:** <https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505>\
**Category:** Elastic Search\
**Created:** [September 13, 2024, 1:18am UTC](https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505 "2024-09-13T01:18:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![FF\_E4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ff_e4/32/137563_2.png) [@FF\_E4](https://discuss.elastic.co/u/FF_E4)\
**Post date:** [September 13, 2024, 1:18am UTC](https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505/1 "2024-09-13T01:18:58Z")

</div>

There are several issues mentioned here.

To start with, I have a setup where there's about 20 different servers who's logs are sent to a central local server I run with elastic stack 7.17. This is a simple 1 shard setup with no replication. Looking at the indexes, I can see over 50+ indices listed in "index management", like Filebeat-7.17.23-2024-8-19, Filebeat-7.17.8-2024-7.6, etc.

**Index Templates**

It appears that each time I installed filebeat on a machine and ran filebeat setup -e, it created an index template on the local ELK server for that filebeat version. So I have these index templates:

## Index Template | Index Pattern

filebeat-7.17.12. | filebeat-7.17.12-\*  
filebeat-7.17.14. | filebeat-7.17.14-\*  
filebeat-7.17.19. | filebeat-7.17.19-\*  
filebeat-7.17.22. | filebeat-7.17.22-\*  
filebeat-7.17.23. | filebeat-7.17.23-\*  
filebeat-7.17.8. | filebeat-7.17.8-\*  
filebeat-7.17.9. | filebeat-7.17.9-\*

**Indices**  
From these, I have 50+ different filebeat indices like:

filebeat-7.17.8-2024.08.23  
filebeat-7.17.8-2024.08.24  
filebeat-7.17.22-2024.08.20-000003

Can I get this more simplified? Like one filebeat index template? Where all these filebeat sources are merged into one?

**ILM**  
I have an ILM policy ("filebeat") created that rolls over when disk space or date considerations are triggered.

However, when I look at the indexes, I will randomly see lifecycle policy errors, and then they go away on their own, then come back, etc. I'm somewhat concerned as I dont know if this is normal behavior or something I need to deal with. When an error comes in it is of this type:

`"illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.17.8] does not point to index [filebeat-7.17.8-2024.08.25]"`

I read up on aliases, attempted to create them, then hit a problem in that I needed a writable index - I guess only one can be writable per alias?

I have a dozen filebeat-7.17.8-2024.8.17, etc.... and I was hoping to make one alias for all of them but evidently using "is\_write\_index" can only be applied as a one to one... I can't apply it to each of the indicies generated:

```auto
POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "filebeat-7.17.8-*",
        "alias": "filebeat-7.17.8-alias",
        "is_write_index": true
      }
    }
  ]
}

```

**Tl;DR**  
How can I:

1. clean this up... all these filebeat indexes - can they be merged into one from ELK's side? Like a single filebeat index template? From that the rollovers are generated?
2. what should I do to setup aliases the proper way? Should I make an alias for each index templates? Hopefully it can be one template, but if not, is the best practice here to have an alias for each of the 7 different index templates I have?
3. For "is\_write\_index": true, is that applied to the index template, or each index rollover generated? If so, how is dynamically applied?
4. ILM errors: I've read a lot about this, attempted solutions, no luck... and I don't know why the error vanishes and then returns again later on. I can have 50 in error, then refresh and it's 10 in error... refresh an hour later and it's 43 in error.

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [September 13, 2024, 4:40am UTC](https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505/2 "2024-09-13T04:40:59Z")

</div>

As I can see you're using different versions of filebeat across the servers which creating different indexes with default name.

1. I would recommend if you can use same filebeat version.
2. You can give same index name / pattern in each filebeat's configuration file but it may cause some issue because you are using different versions.
3. You don't need to create a alias, Once you'll freshly install the Filebeat with same version on all the servers - It use use same Index pattern and you can directly hit query on `filebeat-*`.

---

<div class="post-metadata">

**Author:** ![FF\_E4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ff_e4/32/137563_2.png) [@FF\_E4](https://discuss.elastic.co/u/FF_E4)\
**Post date:** [September 13, 2024, 5:09am UTC](https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505/3 "2024-09-13T05:09:21Z")

</div>

Thanks for the response. I might try renaming the indexes in the filebeat.yml to be the same, I'll need to look up the setting to do that... however, I dont think I have an issue is with the index pattern. I can use filebeat-\* pattern to pull from all the different indexes just fine, to run visualizaitons, etc. My concern is in the ILM, as it seems to throw errors like "illegal\_argument\_exception: index.lifecycle.rollover\_alias [filebeat-7.17.8] does not point to index [filebeat-7.17.8-2024.08.25]"

Getting the indexes all named the same, is a great place to start simplifying I think. But even then, the ILM issue is concerning me. Do you have advice on how to handle these types of errors? Even with the same filebeat name, I think this type of error will persist.

---

<div class="post-metadata">

**Author:** ![FF\_E4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ff_e4/32/137563_2.png) [@FF\_E4](https://discuss.elastic.co/u/FF_E4)\
**Post date:** [September 13, 2024, 8:39am UTC](https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505/4 "2024-09-13T08:39:41Z")

</div>

I started reindexing my indexes that were filebeat-[version]-[date] to filebeat[unique string]-00001. Then I made an alias to it. Removed the alias from the index template. Then attach an ILM to the new index. This seems to get past all the errors I was getting.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2024, 8:40am UTC](https://discuss.elastic.co/t/ilm-need-help-organizing-my-indexes/366505/5 "2024-10-11T08:40:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.