Metricbeat 7.4 index not created daily

[Kawther (Mastouri)]

Hello,

I upgraded metricbeat from 6 to 7.4 .
I have this problem , I realized that the index is only created with the date you started the service

I want to use the old format <beat-type>-<hostname>-<date>

GET _cat/indices/metricbeat
yellow open metricbeat-7.4.0-2020.04.06-000001 kCtjwDQjQRCjDl28XWIpzw 1 1 10250 0 4.4mb 4.4mb

Any idea please ?

Thank's.

[jsoriano (Jaime Soriano)]

Hey @Kawther, welcome to discuss

This is expected, since 7.x indexes are created by index lifecycle management (ILM). ILM creates a single alias (metricbeat-<version>) to where metricbeat sends the data. This alias is configured to point to an index where all the data is written. ILM starts a new index when the current index is older than a month or bigger than 50GB. This is intended, to avoid having too many small indexes, but still have some .
This behaviour is controlled by an index lifecycle policy maintained by Metricbeat, and can be configured.

You can read more about this here: https://www.elastic.co/guide/en/beats/metricbeat/7.6/ilm.html

[Kawther (Mastouri)]

@jsoriano thank you for your answer
I tried to update rollover for my indices with index older to 1 day .

GET _ilm/policy/metricbeat-7.4.0
{
  "metricbeat-7.4.0" : {
    "version" : 55,
    "modified_date" : "2020-04-06T19:03:18.028Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "1gb",
              "max_age" : "1d"
            }
          }
        }
      }
    }
  }
}

But index policy are not realy updated .
So I tried with :

POST /metricbeat-7.4.0/_rollover/
{
  "conditions": {
    "max_age":   "1d",
    "max_size":  "1gb"
  }
} 

Response:

    {
      "acknowledged" : false,
      "shards_acknowledged" : false,
      "old_index" : "metricbeat-7.4.0-2020.04.06-000001",
      "new_index" : "metricbeat-7.4.0-2020.04.06-000002",
      "rolled_over" : false,
      "dry_run" : false,
      "conditions" : {
        "[max_size: 1gb]" : false,
        "[max_age: 1d]" : false
      }
    }

I am not able to update index policies.

[jsoriano (Jaime Soriano)]

@Kawther take a look to these docs:

• https://www.elastic.co/guide/en/elasticsearch/reference/7.6/_updates_to_executing_policies.html
• https://www.elastic.co/guide/en/elasticsearch/reference/7.6/indices-rollover-index.html

You would need to use something like this:

PUT _ilm/policy/metricbeat-7.4.0
{
  "policy" : {
    "phases" : {
      "hot" : {
        "min_age" : "0ms",
        "actions" : {
          "rollover" : {
            "max_size" : "1gb",
            "max_age" : "1d"
          }
        }
      }
    }
  }
}

This wouldn't update current index, but you can manually rollover it, and the next one will have the new policy. To force a rollover:

POST metricbeat-7.4.0/_rollover

You can find what policies are being used by the indexes managed by metricbeat-7.4.0 rollover alias with:

GET metricbeat-7.4.0/_ilm/explain

Take into account that you will need to repeat the process when you upgrade Metricbeat, because the version is included in the policy name. Since 7.6.0 (elastic/beats#14745) the version is not included, so Metricbeat continues using custom policies if defined.

[Kawther (Mastouri)]

@jsoriano Thank you very much, you solved my problem