Why I am getting one huge metricbeat index instead of daily indices?

[elasticheart]

Hi,

I am using elastic stack 7.9.2. I have set up metricbeat and it is exporting to elasticsearch. I haven't modified the index name setting in metricbeat.yml and is taking the default setting index: "metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}".

What I expected was metricbeat index getting created daily, like metricbeat-7.9.2-2020.10.1 , metricbeat-7.9.2-2020.10.2 , metricbeat-7.9.2-2020.10.3 etc. Also, I was thinking about creating some ILM policy, so that indices older than 5 days will get automatically deleted.

Now, when I take a look at the indices, I can see one huge index metricbeat-7.9.2-2020.10.14-000001 (started metricbeat on 14th). Why its happening like this and how can I fix this, to achieve what I am looking for?

Thanks.

[borna_talebi]

Hi,
In your metricbeat.yml file did you disabled ILM using setup.ilm.enabled: false?
Because if you didn't, metricbeat will ignore index: "metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}" and use indices with numbers at the end to manage the lifecycle.

according to docs:

When index lifecycle management (ILM) is enabled, the default index is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}" , for example, "filebeat-7.9.2-2020-10-21-000001" . Custom index settings are ignored when ILM is enabled.

[elasticheart]

Hi @borna_talebi , below is my ILM config

# index name.
#setup.ilm.enabled: auto
#setup.ilm.rollover_alias: 'metricbeat'
#setup.ilm.pattern: "{now/d}-000001"
#setup.ilm.policy_name: "mypolicy"
#setup.ilm.policy_file:
setup.ilm.check_exists: false
#setup.ilm.overwrite: false

I have just disabled setup.ilm.check_exists and everything else is default. Is there anything need to be modified?

[borna_talebi]

@elasticheart

Is there anything need to be modified?

It depends. if you want to fix your huge index then don't change anything and just configure your ILM and it should fix it.
But if you want to have daily indices then use setup.ilm.enabled: false

[elasticheart]

@borna_talebi I got it and thanks for your suggestion

Can you advice me how to configure ILM, say if I just want 1 index metricbeat, but I don't want messages older than 5 days in it. What should I do to achieve this? I don't want to archive old indices, say I don't want warm and cold. I just want the messages older than 5 days to get deleted. Also, I would like to know at what time this policy apply on the indices, say, the message on 01-10-2020 get deleted at 12'o clock on 16-10-2020?

[borna_talebi]

@elasticheart

but I don't want messages older than 5 days in it

I'm not sure if this is the best solution but here is my suggestion:

1. In hot phase, set rollover to 1 day so ilm would create daily indices.

2. In delete phase, set Timing for delete phase to 5 days from rollover.

Just to be clear, ILM works with indices not messages(documents).

[elasticheart]

Ok. Lemme try out this and I will reply here

[elasticheart]

I did whatever need to be, still no hope. Lemme explain;

I have these many indices

And this is my metricbeat index I am concerned about

This is my ILM policy

These are my templates

And this is the template explained

Below is my ILM policy explained.

PUT _ilm/policy/metricbeat
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "1gb"
          },
          "set_priority": {
            "priority": null
          }
        }
      },
      "warm": {
        "actions": {
          "set_priority": {
            "priority": null
          }
        }
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

As per the policy, the index should be max 1 GB size and 1 day old. This index was created yesterday and more than 1 GB size (pri 1.1GB, total 2.1GB). How to fix this?