Rollover Indeces using Logstash and Elastic

Elastic Stack Logstash
1

Hi guys,

I have following use case:

I want to create new index every day, therefore my logstash output configuraiton looks as this:

output {
  elasticsearch {
	index => "metricbeat-linux-vms-%{+YYYY.MM.dd}"

I also have an ILM policy which defines rollover after 50gb and I have an index template which uses this ilm policy and is applied to all indeces with metricbeat-linux-vms*.

Where do I define that after 50gbs are reached a new index is created like: metricbeat-linux-vms-2020-07-31-000001?

Thank you very much

2

We aren't all guys :slight_smile:

What does your ILM policy look like?

3

Thank you for your response.

Sorry, by saying "guys" I do not have specific gender in my mind. So my ILM policy is pretty easy: I want to create a new index onces my existing one has reached 50GB. I created in in Kibana. And I have an Indx template, which is using this policy:

{
  "index": {
    "lifecycle": {
      "name": "ilm-metricbeat-my-policy"
    }

But my struggle is: I have daily indeces. Every day I create a new index by using this configuration in logstash output:

index => "metricbeat-linux-vms-%{+YYYY.MM.dd}"

What I don't understand is: How can I define the creation of new index after 50 GB like

"metricbeat-linux-vms-%{+YYYY.MM.dd}-000001"

So during the day I have following indeces:

"metricbeat-linux-vms-2020.08.04"
"metricbeat-linux-vms-2020.08.04-000001"
"metricbeat-linux-vms-2020.08.04-000002"

next day:

"metricbeat-linux-vms-2020.08.05"

4

That's not now ILM is designed. The timestamp in the index name is when the policy was first used.

5

So how would you do a rollover on daily indeces ?

6

You define it in the policy as max_age.

7

What would be the rollover alias in this case? I need to provide it in the index template and I am not sure what I need to provide here.

So you would suggest to leave out the timestamp from the index name and to have instead an index like:

metricbeat-linux-vms ?

I have problems to understand how I can create daily indeces then. My idea was to create an index every day to maintain them easier (for example delete all index with metricbeat-linux-vms-2020.08*"

8

The idea is that ILM knows how old the indices are, even if they aren't created with a date in the name. So all you need to do there is set the retention period in the policy.

9

Ok. Thanks.

Could you please explain what the desired approach would be like using filebeat -> logstash -> elasticsearch for that?

As far as I understood:

In filebeat I have:

output.logstash:
  hosts: ["logstash.host:5044"]

In logstash I have:

output {
  elasticsearch {
    hosts => ["https://elastic.host:9200"]
	index => "metricbeat-linux-vms"
  }
}

I have a policy ilm-metricbeatwith max_age: 1d and enabled rolle over

I have an index template:

{
  "index": {
    "lifecycle": {
      "name": "ilm-metricbeat",
      "rollover_alias":  "metricbeat-linux-vms"

    }
}

This configuration force the creation of new index after 1d has passed. Am I correct so far?

Things I do not understand yet:

  1. How can I distiguishe the index created yesterday from the one created today, since they have the same name
  2. How I can access the logs from yesterday, when a new index was created today?
  3. How can I use variables in the index name, when I decide later to seperate my logs or metrics into multiple indeces?
  4. How can I acess the indeces created, let's say during the last month to backup them?

Thank you

10

If that is what you have in the policy under max_age, then yes.

  1. The numeric prefix will change, starting from 000001 . So each day will increment by +1
  2. Access them how?
  3. You split them out and use different ILM policies for them
  4. Use SLM
11

The numeric prefix will change, starting from 000001 . So each day will increment by +1

Where do you define that? in the logstash pipeline configuration or on the ilm configuration or in the index template?

12

It's built in behaviour.

13

What is inside the rollover_alias of the index template then ? Just 1:1 the name of the index ?

So when I want to write into index: metricbeat-linux-vms my rollover_alias will be metricbeat-linux-vms and when I set the max_age to 1 day after a day has passed I will see an index called:

metricbeat-linux-vms

and one called

metricbeat-linux-vms-00001

Is it correct?

14

That is the alias, so it's not an index.

That would be the very first index that is created that is attached to the above alias.

The next day's index would be metricbeat-linux-vms-00002, and so on.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.