Filebeat module custom policy management

Elastic Stack Beats
1

I want to apply custom ilm_policy for every index that I create with filebeat.

I tried below approach

`#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["someip:9200"]
  username: "elastic"
  password: "somepassword"
  ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca/ca.crt"]
  ssl.certificate: "/etc/elasticsearch/certs/filebeat/filebeat.crt"
  ssl.key: "/etc/elasticsearch/certs/filebeat/filebeat.key"
  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  indices:
    - index: "filebeat-netflow-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "netflow"

    - index: "filebeat-cisco-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "cisco"


#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
script.max_compilations_rate: 2000/5m
script.cache_max_size: 1000


setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-cisco"
setup.ilm.pattern: "{now/d}-rolled"
setup.ilm.check_exists: true
setup.ilm.policy_file: /etc/filebeat/ilm_policy_cisco.txt
setup.ilm.overwrite: true

setup.ilm.rollover_alias: "filebeat-netflow"
setup.ilm.pattern: "{now/d}-rolled"
setup.ilm.check_exists: true
setup.ilm.policy_file: /etc/filebeat/ilm_policy_netflow.txt
`

Policy in /etc/filebeat/ilm_policy_netflow.txt

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "200gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "cold": {
        "min_age": "2d",
        "actions": {
          "set_priority": {
            "priority": 25
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This obviously only applies only the last configured value, not per index. One weird thing is that rolling occurs immediately as shown below.

image
image1371×121 7.7 KB

How can I apply different policy file per index?
Why the file is rolled immediately after being created?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.