[Solved] Indexes does not automatically created based on new date

[TsuWeiQuan]

Hello,

I have this small problem where I realized my beats are not creating new indexes as the day goes by. All data are stored in a single index which doesn't look good and can be a problem it was deleted.

This problem arose when I changed my index name on the beats setting.
As I prefer the naming format to be <beat-type>-<hostname>-<date> which is shown in the picture below.

All beats are sending logs directly into elasticsearch via modules and i have append these settings in each beat's yml file.

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat-esnode1"
setup.ilm.pattern: "{now/d}-000001"
setup.template.overwrite: true

As you can see from the above picture, the metricbeat index contains 1.9Gb of data from 5 hosts with the date created on 23 May.

Is there a way to make these indexes create a new index after each day has passed Or this is actually the ideal way to be done?

[faec (Fae Charlton)]

Hi! Have you configured an index lifecycle policy, as described in these docs? If so, could you share your configuration?

[TsuWeiQuan]

I did not configure the index lifecycle policy, leaving them in the default state.
Thank you for mentioning this, i did not know the purpose of ILM until now.
PS i am still a newbie in elastic

After reading the Documentation, i understand that the indices will only be updated to the lastest after a phase change. However, the default state that was set was max_size: 50gb & "max_age": "30d" in the hot phase, version 32.

So I followed the documentation to do an update the policies while indices are still writing.
However, after countless of tries, running GET filebeat-*/_ilm/explain still shows the indices version at 32.

{
  "indices" : {
    "filebeat-esnode2-2019.05.29-000001" : {
      "index" : "filebeat-esnode2-2019.05.29-000001",
      "managed" : true,
      "policy" : "filebeat-7.0.1",
      "lifecycle_date_millis" : 1559118013908,
      "phase" : "hot",
      "phase_time_millis" : 1559118014150,
      "action" : "rollover",
      "action_time_millis" : 1559118107765,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1559118107765,
      "phase_execution" : {
        "policy" : "filebeat-7.0.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 32,
        "modified_date_in_millis" : 1559118013357
      }
    },
    "filebeat-esnode1-2019.05.29-000001" : {
      "index" : "filebeat-esnode1-2019.05.29-000001",
      "managed" : true,
      "policy" : "filebeat-7.0.1",
      "lifecycle_date_millis" : 1559120066378,
      "phase" : "hot",
      "phase_time_millis" : 1559120066554,
      "action" : "rollover",
      "action_time_millis" : 1559120507394,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1559120507394,
      "phase_execution" : {
        "policy" : "filebeat-7.0.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 34,
        "modified_date_in_millis" : 1559120065826
      }
    }

Q1) Is it true that I have no way to change the settings until the criteria for rollover has been met?

Q2) Is there a recommended policy practice such as 2Gb/Index with a 90 days warm phase before deletion?

[TsuWeiQuan]

I have managed to solve this question via another post of mine.

Thank you!