ILM: Need help organizing my indexes

[FF_E4]

There are several issues mentioned here.

To start with, I have a setup where there's about 20 different servers who's logs are sent to a central local server I run with elastic stack 7.17. This is a simple 1 shard setup with no replication. Looking at the indexes, I can see over 50+ indices listed in "index management", like Filebeat-7.17.23-2024-8-19, Filebeat-7.17.8-2024-7.6, etc.

Index Templates

It appears that each time I installed filebeat on a machine and ran filebeat setup -e, it created an index template on the local ELK server for that filebeat version. So I have these index templates:

Index Template | Index Pattern

filebeat-7.17.12. | filebeat-7.17.12-*
filebeat-7.17.14. | filebeat-7.17.14-*
filebeat-7.17.19. | filebeat-7.17.19-*
filebeat-7.17.22. | filebeat-7.17.22-*
filebeat-7.17.23. | filebeat-7.17.23-*
filebeat-7.17.8. | filebeat-7.17.8-*
filebeat-7.17.9. | filebeat-7.17.9-*

Indices
From these, I have 50+ different filebeat indices like:

filebeat-7.17.8-2024.08.23
filebeat-7.17.8-2024.08.24
filebeat-7.17.22-2024.08.20-000003

Can I get this more simplified? Like one filebeat index template? Where all these filebeat sources are merged into one?

ILM
I have an ILM policy ("filebeat") created that rolls over when disk space or date considerations are triggered.

However, when I look at the indexes, I will randomly see lifecycle policy errors, and then they go away on their own, then come back, etc. I'm somewhat concerned as I dont know if this is normal behavior or something I need to deal with. When an error comes in it is of this type:

"illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.17.8] does not point to index [filebeat-7.17.8-2024.08.25]"

I read up on aliases, attempted to create them, then hit a problem in that I needed a writable index - I guess only one can be writable per alias?

I have a dozen filebeat-7.17.8-2024.8.17, etc.... and I was hoping to make one alias for all of them but evidently using "is_write_index" can only be applied as a one to one... I can't apply it to each of the indicies generated:

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "filebeat-7.17.8-*",
        "alias": "filebeat-7.17.8-alias",
        "is_write_index": true
      }
    }
  ]
}

Tl;DR
How can I:

1. clean this up... all these filebeat indexes - can they be merged into one from ELK's side? Like a single filebeat index template? From that the rollovers are generated?
2. what should I do to setup aliases the proper way? Should I make an alias for each index templates? Hopefully it can be one template, but if not, is the best practice here to have an alias for each of the 7 different index templates I have?
3. For "is_write_index": true, is that applied to the index template, or each index rollover generated? If so, how is dynamically applied?
4. ILM errors: I've read a lot about this, attempted solutions, no luck... and I don't know why the error vanishes and then returns again later on. I can have 50 in error, then refresh and it's 10 in error... refresh an hour later and it's 43 in error.

[ashishtiwari1993 (Ashish Tiwari)]

As I can see you're using different versions of filebeat across the servers which creating different indexes with default name.

1. I would recommend if you can use same filebeat version.
2. You can give same index name / pattern in each filebeat's configuration file but it may cause some issue because you are using different versions.
3. You don't need to create a alias, Once you'll freshly install the Filebeat with same version on all the servers - It use use same Index pattern and you can directly hit query on filebeat-*.

[FF_E4]

Thanks for the response. I might try renaming the indexes in the filebeat.yml to be the same, I'll need to look up the setting to do that... however, I dont think I have an issue is with the index pattern. I can use filebeat-* pattern to pull from all the different indexes just fine, to run visualizaitons, etc. My concern is in the ILM, as it seems to throw errors like "illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.17.8] does not point to index [filebeat-7.17.8-2024.08.25]"

Getting the indexes all named the same, is a great place to start simplifying I think. But even then, the ILM issue is concerning me. Do you have advice on how to handle these types of errors? Even with the same filebeat name, I think this type of error will persist.

[FF_E4]

I started reindexing my indexes that were filebeat-[version]-[date] to filebeat[unique string]-00001. Then I made an alias to it. Removed the alias from the index template. Then attach an ILM to the new index. This seems to get past all the errors I was getting.