Massive index creation after conditional indexing

Elastic Stack Beats
,
1

We have a server, where we had previously configured a filebeat for a simple monitoring of log files. A month ago we were asked to ingest the record of a containered docker application, but this had to be recorded in a separate index.
We had two options, set conditional indexing in the same filebeat implementation we had at the time, or set a filebeat container docker for this purpose. As we had implemented filebeat previously, we set conditional indexing in the same filebeat implementation we had at the time.

From day one, the filebeat is creating an index per day, and at this time, with the previous record of the container (which was also recorded, creating indexes with the registration date of those days. I mean, this was configured 2024.04.19 and elastic created indexes from 2024.02.20, day by day), we have 103 indexes of this monitoring. Besides the daily indexes from the previous monitoring.

Here you have the filebeat configuration file, ilm policies jsons, and evidences of index daily creation of both service monitoring and filebeat base monitoring.

evidences
evidences1920×1080 125 KB

2

Can you provide some context of what is the issue here?

You are telling filebeat to create an index per day per filebeat and in some cases an index per day per filebeat and per server name.

So it is expected that if you have for example 10 filebeats and 10 servers, you will have hundreds of indices per day.

Not exactly sure what is the problema here.

3

The problem is i dont want an index per day. Where am i telling filebeat to create an index per day?

4

I can't copy because you shared as image, but in your filebeat configuration in the elasticsearch output block your index setting ends with %{+yyyy.MM.dd}.

This will tell filebeat to use the date as part of the index name which means that it will create an index per day.

If you do not want this, you need to remove this part.

5

Thanks for ur help and time. I changed that parameter in the configuration on both indexes and the issue is solved. But another issue is still there, the second index, the docker's one, is not configured as the base's one. The policy isn't been applied, but I don't know why, because I'm not able to configure an independent ilm and template configuration for it. I have tried manually configuring a new template cloning the other one, and doing the same with the ilm policy, but doesn't work at all.
What am i missing here?

6

I do not use filebeat to set templates, but if I'm not wrong you can set only one template in filebeat.

Not sure it would be possible to set the template using filebeat in your case since the name is dynamic.

I would recommend to set the template manually and make sure that the index pattern in the template will match your desired indices.

7

I tried, but the problem now is not with the template itself. The problem is that the docker index alias is not configured automatically. Despite this, I configure it manually and having the ilm directive configured, and the action applied , the rollover is not done.

8

Oh yeah, you also can have only one alias per template.

The main issue here is that you want to have the name of the server in the index name, and to use rollover in this case you need to have one template per server, you cannot have dynamic rollovers for example.

Can you share your template? In plain text please, do not share it is image. Use the Preformatted text option, the </> button, to share it.

9

No, is not the server's index, is a service's index. What im undertanding about what u are suggesting me is this. Previus the filebeat starts ingesting data in the second index i have to create a index template and a ilm policy, and then from the template create the index where the filebeat will ingest the data. Am i wrong?

10

It doesn't matter if it is a server or a service, if it is a dynamic value you need 1 template per value.

For example, If you have serviceA, serviceB and serviceC, you will need one template that will match the index name for each one of those services.

The rollover alias cannot be dynamic, so if you want ILM to rollover your data, then you need one alias per index.

11

Okey but how can i, in my case, tell filebeat to use the service index template in the service index? Since is only possible to configure a single template in the configuration.

12

I had first created index templates and ilm policies manually, and then try both via filebeat and manually create indices. But now the ilm doesnt work.
How can i configure the filebeat output to always point to the new index if i want to create new indices with the date and number "{now/d}-000001" of the creation date of the index? Is this posible.
All i want is to send the output to elasticsearch creating indices 100gb max size, and then rotate those ones with the date of the index and number, in the case we generate more than 100GB of log a day in any point (this happened in the past).

13

You need to for example make your indices weekly not daily. After that, you can clone the default index template and change its index pattern to match better different indexes. After that, you can set up different ILM policies on the template.