How properly rollover indices

[dorinand]

I harvest logs with filebeat from all docker containers, sending them to logstash and from logstash are forwarded to elasticsearch. I would like to rollover my indices, that are automatically created, if they are too big or too old. There are few variants I tested.

1. Use Elasticsearch ILM. I created policy my_policy1 that should rollover when 5M are exceeded and template template_charizard:
   {
   "index_patterns" : ["printmessage_charizard*"],
   "settings" : {
   "number_of_shards" : 1,
   "number_of_replicas": 1,
   "index.lifecycle.name": "my_policy1",
   "index.lifecycle.rollover_alias": "charizard-actual"
   },
   "aliases": {
   "charizard-actual": {
   "is_write_index": true
   }
   }
   }

   This does not work, I have to create index with alias , in this case charizard-actual manualy. If I not, there will be error:

   Rollover alias [charizard-actual] can point to multiple indices, found duplicated alias [[charizard-actual]] in index template [template_charizard]

   There will be more container that will be consumed, so I cant manually create index every time when some new index should be created. I would expect that it will be create automatically.

   UPDATE:

   I tested this option (exactly by tutorial, with the same names), and it does not work. It just create new empty index, with increment number, but all logs went to the bootstrap index.

2. I can configure logstash ILM. This works fine, but I cant use variables in alias name, so I would have to create elasticsearch output for each index and make a lot of ifs.

3. I can configure filebeat ILM. Similar problem like with logstash. I can use variable in rollover alias, but it is hardcoded for all indices.

My expectation is, that I will configure some policies, some templates, and based on regexp of index name will be template applied to the index. All the options are basically usefull, everytime there must be manual change (edit logstash config) or request (create index manually first).

Questions:
What is the best solution for proper rollover?
Why everything must be hardcoded, it is very hard to maintain then.
According to me, curator would be the best solution, but then tehere is neccessitiy to maintain another application.
Could anybody explain me usage of all ILM options? Cant imagine to change configuration for every new index, even create manually index first or use hardcoded alias for all logs like in filebeat example. Maybe I am missing something.

[ming1997 (Vector_Ming)]

At first, your index name should be formatted, like filebeat-000001 or filebeat-2020.06.17-000001.If you use the second formatted index name, the "datetime-part" will automatic change when rollover.
Next, do not add aliase to your template, It will add aliase to your new index which will lead to the "multiple indices" problem.
Then create your initial index maybe like this:

PUT filebeat-2020.06.17-000001
{
  "aliases": {
    "charizard-actual": {
      "is_write_index": true
    }
  }
}

rewrite your logstash elasticsearch output , change the index to "charizard-actual".
Then it will rollover when the index size is more than 5M.And logstash will automatic output to index filebeat-2020.06.17-000002. Just because of you have set output to charizard-actual and ES has set the 000001 index "is_write_index" to false, the 000002 index "is_write_index" to true.

There is one thing to notice that elasticsearch will detect the status of index every 10 minutes(by default). So even if your rollover policy is not over 5M. The actually , single index will be 10M or more. This is my test which I set it not over 1G.

[dorinand]

Hi, thanks for your reply. From your explanation, I assume you are react to my point n.3 - ILM with Filebeat.

I know, I can change the name of index. But my Filebeat harvest from more images, that means, there will be more indices, based on image name. Then, I cant use this option, because everything would be pushed to the same index, that would lead to errors.

This is my expectation. I am running two containers printmessage_charizard_1 and printmessage_pikachu_1, later there will be more containers, some developers will deploy new app, lets say printmessage_charmander_1, etc... I would like to have indices rollovered based on size, and named based on container name, so in the end, this would be my indices:

printmessage_charizard_1-000001
printmessage_charizard_1-000002
printmessage_charizard_1-000003
printmessage_charizard_1-000004
...
printmessage_pikachu_1-000001
printmessage_pikachu_1-000002
printmessage_pikachu_1-000003
...

My problem with filebeat and also logstash is the same. Name of index would be the same for all indices. I cant change it based od container name, at least, I cant do it in filebeat. Logstash could be configured based on the container name, but that means, I have to configure elasticsearch output for all indices - too hard to maintain, connections to elastic would increase with number of indices. Elasticsearch ILM enables it, but I have to create each index manually first, again, too complicated.

Your explanation relies on fact, there is one filebeat per application.

My goal: everybody who will create new app and their logs will be harvested by filebeat would have indexes that are rollovered automatically based on policy. So basicaly I would not have to change configuration of filebeat, logstash or make manual requests to elasticsearch everytime somebody would like to have index in ELK.