[Solved] Indexes does not automatically created based on new date

Hello,

I have this small problem where I realized my beats are not creating new indexes as the day goes by. All data are stored in a single index which doesn't look good and can be a problem it was deleted.

This problem arose when I changed my index name on the beats setting.
As I prefer the naming format to be <beat-type>-<hostname>-<date> which is shown in the picture below.

All beats are sending logs directly into elasticsearch via modules and i have append these settings in each beat's yml file.

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat-esnode1"
setup.ilm.pattern: "{now/d}-000001"
setup.template.overwrite: true

As you can see from the above picture, the metricbeat index contains 1.9Gb of data from 5 hosts with the date created on 23 May.

Is there a way to make these indexes create a new index after each day has passed Or this is actually the ideal way to be done?

Hi! Have you configured an index lifecycle policy, as described in these docs? If so, could you share your configuration?

I did not configure the index lifecycle policy, leaving them in the default state.
Thank you for mentioning this, i did not know the purpose of ILM until now.
PS i am still a newbie in elastic :slight_smile:

After reading the Documentation, i understand that the indices will only be updated to the lastest after a phase change. However, the default state that was set was max_size: 50gb & "max_age": "30d" in the hot phase, version 32.

So I followed the documentation to do an update the policies while indices are still writing.
However, after countless of tries, running GET filebeat-*/_ilm/explain still shows the indices version at 32.

{
  "indices" : {
    "filebeat-esnode2-2019.05.29-000001" : {
      "index" : "filebeat-esnode2-2019.05.29-000001",
      "managed" : true,
      "policy" : "filebeat-7.0.1",
      "lifecycle_date_millis" : 1559118013908,
      "phase" : "hot",
      "phase_time_millis" : 1559118014150,
      "action" : "rollover",
      "action_time_millis" : 1559118107765,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1559118107765,
      "phase_execution" : {
        "policy" : "filebeat-7.0.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 32,
        "modified_date_in_millis" : 1559118013357
      }
    },
    "filebeat-esnode1-2019.05.29-000001" : {
      "index" : "filebeat-esnode1-2019.05.29-000001",
      "managed" : true,
      "policy" : "filebeat-7.0.1",
      "lifecycle_date_millis" : 1559120066378,
      "phase" : "hot",
      "phase_time_millis" : 1559120066554,
      "action" : "rollover",
      "action_time_millis" : 1559120507394,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1559120507394,
      "phase_execution" : {
        "policy" : "filebeat-7.0.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 34,
        "modified_date_in_millis" : 1559120065826
      }
    }

Q1) Is it true that I have no way to change the settings until the criteria for rollover has been met?

Q2) Is there a recommended policy practice such as 2Gb/Index with a 90 days warm phase before deletion?

I have managed to solve this question via another post of mine.

Thank you!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.