Filebeat is not attaching lifecycle policy to indexes created

Elastic Stack Beats
,
1

I use filebeat as a docker container in our eks cluster, which travels through private DNS to our another eks cluster which hosts ECK stack, i have setup everything the indexes are created the logs are pushing , data streams and views are setup. I want to setup 7 days index retention policy which is created already in kibana dashboard.

The issue is that i'm unable to attach the policy to each index created the logs policy only gets attached to filebeat index template only not to the other dev or prod indexes created.

Here is my

filebeat.yml

filebeat.config:
  modules:
    path: /usr/share/filebeat/modules.d/*.yml
    reload.enabled: false

processors:
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"
      match_source: true
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
  - drop_fields:
      fields: 
        - "@timestamp"
        - "message.keyword"
        - "agent.ephemeral_id"
        - "agent.ephemeral_id.keyword"
        - "agent.id"
        - "agent.id.keyword"
        - "agent.name"
        - "agent.name.keyword"
        - "agent.type"
        - "agent.type.keyword"
        - "agent.version"
        - "agent.version.keyword"
        - agent
        - "ecs.version"
        - "host.name"
        - "input.type"
        - "log.file.path"
        - "log.offset"
        - "stream"
        - "_id"
        - "_index"
        - "_version"
        - "_score"
        - "fields"
        - "sort"

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/nbula-fe-*.log
  index: "${var.is_dev ? "dev" : "prod"}-nbula-fe-dev-%%{+yyyy.MM.dd}"
  lifecycle:
    name: "${var.is_dev ? "logs-policy-dev" : "logs-policy-prod"}"
    rollover_alias: ["rollover-${var.is_dev ? "dev" : "prod"}"]

- type: container 
  paths:
    - /var/log/containers/nbula-be-*.log
  index: "${var.is_dev ? "dev" : "prod"}-nbula-be-dev-%%{+yyyy.MM.dd}"
  lifecycle:
    name: "${var.is_dev ? "logs-policy-dev" : "logs-policy-prod"}"
    rollover_alias: ["rollover-${var.is_dev ? "dev" : "prod"}"]

- type: container
  paths:
    - /var/log/containers/nbula-ai-*.log  
  index: "${var.is_dev ? "dev" : "prod"}-nbula-ai-dev-%%{+yyyy.MM.dd}"
  lifecycle:
    name: "${var.is_dev ? "logs-policy-dev" : "logs-policy-prod"}"
    rollover_alias: ["rollover-${var.is_dev ? "dev" : "prod"}"]

- type: container
  paths:
    - /var/log/containers/nbula-mm-*.log
  index: "${var.is_dev ? "dev" : "prod"}-nbula-mm-dev-%%{+yyyy.MM.dd}"
  lifecycle:
    name: "${var.is_dev ? "logs-policy-dev" : "logs-policy-prod"}"
    rollover_alias: ["rollover-${var.is_dev ? "dev" : "prod"}"]

- type: container
  paths:
    - /var/log/containers/nxus-ls-app-*.log
  index: "${var.is_dev ? "dev" : "prod"}-nxus-ls-app-dev-%%{+yyyy.MM.dd}"
  lifecycle:
    name: "${var.is_dev ? "logs-policy-dev" : "logs-policy-prod"}"
    rollover_alias: ["rollover-${var.is_dev ? "dev" : "prod"}"]

output.elasticsearch:
  hosts: ["${var.elastic_domain}"]
  ssl.enabled: true
  username: "${var.elastic_username}"
  password: "${var.elastic_password}"

setup.ilm.enabled: true
setup.template.overwrite: true
setup.template.settings:
  index.lifecycle.name: "${var.is_dev ? "logs-policy-dev" : "logs-policy-prod"}"
  index.lifecycle.rollover_alias: "rollover-${var.is_dev ? "dev" : "prod"}"

kibana:
  enabled: true
  host: "${var.kibana_domain}"
  ssl:
    enabled: true
  username: "${var.elastic_username}"
  password: "${var.elastic_password}"

Please guide me what im doing wrong?

End goal is to have daily indexes of dev and prod services and dev logs should be retained each 7 days and prod for 30 days.

P.S the vars will be substituted.

Thanks

2

Hi @Saif_Ali, Welcome to the Elastic community.

This more looks env vars issue. Can you check whether the policy is created or not ? also can you just pass the static ILM in your configuration and check.