ILM setting are not honored

Hi,

We have configured our filebeat indices configure to make rollover when they reach 50G or every day. The 50G condition is never honored, the daily rollover is not done every day. There are no other changes in the default configuration so the ILM should be checked every 10min (), Are we doing something wrong?

  "name": "kibana-kibana-7d47997578-qpzrx",
  "uuid": "87fc2b43-9788-4b5f-8739-b50e0892f191",
  "version": {
    "number": "8.0.0",
    "build_hash": "b66af16153eea145b9dde5651b571c95a29e88b7",
    "build_number": 29297,
    "build_snapshot": true
  },

{
  "filebeat-8.0.0" : {
    "version" : 2,
    "modified_date" : "2019-09-20T10:06:56.183Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "15d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "1d",
          "actions" : { }
        }
      }
    }
  }
}


I notice Version 8.0? That's not released yet is it?

The 50G size applies to the primary shards only. If you have 1 replica, the stats will show about 100G when rollover happens and I see some 100G indices.

I agree, something is missing the daily rollover. What template(s) apply to these indices? Check them in increasing order value, is this the only ILM policy referenced? Do you have any other ILM policies?

I notice Version 8.0? That's not released yet is it?

No, it is our snapshot test cluster

What template(s) apply to these indices?

the one created by filebeat

GET _template/filebeat-*

{
  "filebeat-8.0.0" : {
    "order" : 1,
    "index_patterns" : [
      "filebeat-8.0.0-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "filebeat",
          "rollover_alias" : "filebeat-8.0.0"
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_routing_shards" : "30",
        "query" : {
          "default_field" : [
            "message",
            "tags",
            "agent.ephemeral_id",

    ...

              ]
            }
          }
        },
        "mappings" : {
          "_meta" : {
            "beat" : "filebeat",
            "version" : "7.5.0"
          },
          "dynamic_templates" : [
            {
              "labels" : {
                "path_match" : "labels.*",
                "mapping" : {
                  "type" : "keyword"
                },
                "match_mapping_type" : "string"
              }
            },

    ...

          },
          "date_detection" : false
        },
        "aliases" : { }
      }
    }

Check them in increasing order value, is this the only ILM policy referenced?

It's the only ILM that apply to that indices

Do you have any other ILM policies?

yes, we have for APM, Metricbeat, Auditbeat, ... all have the same issue about the daily rotation

{
  "apm-8.0.0-error" : {
    "version" : 2,
    "modified_date" : "2019-10-23T11:47:49.218Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "16d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "readonly" : { },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  },
  "apm-8.0.0-metric" : {
    "version" : 2,
    "modified_date" : "2019-10-23T11:48:01.174Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "9d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "15d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "readonly" : { },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  },
  "apm-8.0.0-span" : {
    "version" : 2,
    "modified_date" : "2019-10-23T11:48:13.978Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "15d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "readonly" : { },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  },
  "apm-8.0.0-transaction" : {
    "version" : 2,
    "modified_date" : "2019-10-23T11:48:33.328Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "15d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "readonly" : { },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  },
  "apm-rollover-30-days" : {
    "version" : 18,
    "modified_date" : "2019-11-25T04:32:21.728Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "warm" : {
          "min_age" : "30d",
          "actions" : {
            "readonly" : { },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  },
  "auditbeat" : {
    "version" : 2,
    "modified_date" : "2019-12-02T04:31:53.947Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "auditbeat-8.0.0" : {
    "version" : 1,
    "modified_date" : "2019-09-20T09:39:17.585Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "filebeat" : {
    "version" : 2,
    "modified_date" : "2019-12-02T04:33:07.725Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "filebeat-7.5.0" : {
    "version" : 1,
    "modified_date" : "2019-11-20T15:56:23.035Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "filebeat-8.0.0" : {
    "version" : 3,
    "modified_date" : "2019-12-13T13:46:45.850Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "8h"
            }
          }
        },
        "delete" : {
          "min_age" : "15d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "1d",
          "actions" : { }
        }
      }
    }
  },
  "heartbeat" : {
    "version" : 3,
    "modified_date" : "2019-12-02T04:32:36.025Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "heartbeat-8.0.0" : {
    "version" : 2,
    "modified_date" : "2019-09-20T10:06:39.877Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "20d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "1d",
          "actions" : { }
        }
      }
    }
  },
  "metricbeat" : {
    "version" : 6,
    "modified_date" : "2019-12-02T04:32:37.920Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "metricbeat-7.3.0" : {
    "version" : 8,
    "modified_date" : "2019-12-13T12:02:01.376Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "10d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : { }
            }
          }
        }
      }
    }
  },
  "metricbeat-8.0.0" : {
    "version" : 5,
    "modified_date" : "2019-10-23T11:27:18.293Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "15d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : { }
        }
      }
    }
  },
  "slm-history-ilm-policy" : {
    "version" : 1,
    "modified_date" : "2019-09-20T09:38:35.352Z",
    "policy" : {
      "phases" : {
        "delete" : {
          "min_age" : "60d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  },
  "watch-history-ilm-policy" : {
    "version" : 1,
    "modified_date" : "2019-09-20T09:38:35.152Z",
    "policy" : {
      "phases" : {
        "delete" : {
          "min_age" : "7d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.