How to setup ILM for filebeat & metricbeat?

[Siavash_Fazli]

Hi Everyone.

I need to set ILM for some beat services.
As you know saving beats docs from the index changed to Datastream in ELK version 8.

I could enable index rollup for heartbeat From Stack Management > Index Lifecycle Policies by these settings:

• Maximum primary shard size: 10GB

• Maximum age: 1d

Now heartbeat indices are separated daily.

I applied the same settings and parameters for filebeat and metricbeat but it didn't work.

This is the filebeat yml configuration:

###################### Filebeat Configuration Example #########################


# ============================== Filebeat inputs ===============================

filebeat.inputs:


- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


setup.dashboards.index: "filebeat-*"
#setup.template.enabled: false
output.elasticsearch.index: "filebeat-gameplatform-%{[agent.version]}"
setup.template.name: "filebeat-gameplatform-%{[agent.version]}"
setup.template.pattern: "filebeat-gameplatform-%{[agent.version]}"
setup.template.overwrite: false
setup.template.fields: "/etc/filebeat/fields-gameplatform.yml"

#setup.ilm.enabled: true
# ================================== KIbana===================================

setup.kibana:
  host: "https://********:443"
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://************:443"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "*******"
  password: "********"
# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

and the same Elasticsearch template setting for metricbeat.

could these settings cause a conflict or overwrite on ILM settings??

[emilioalvap (Emilio Alvarez Piñeiro)]

Hi @Siavash_Fazli,

Thanks for letting us know about your problem.

The first step would be to find ILM policies that you want to update, some of these can be managed and will probably not be updated from beats config, unless forced to do so.

You can start by locating the data streams of interest. ILM applied should appear on the detail section:

There might several policies that apply to different filebeat/metricbeat data streams.

2. Once you know the ILM, you can check the status on Stack Management -> Index Lifecycle Policies. Make sure to check "Include managed system policies":

If it's "Managed", we generally do not recommend editing these directly as changes upstream will likely override your custom options. Instead, you can follow this guide to create a custom policy that overrides the ILM setting.

If it's not managed, then you should be able to edit it in place or create a new one that overrides it. You'll need to manually rollover the data stream to see the changes reflected asap.

Since you have access to the beat config as well, it's possible to configure these manually and override them following these two guides:

• Metricbeat: Configure index lifecycle management.
• Filebeat: Configure index lifecycle managementedit.

I hope this helps.

[Siavash_Fazli]

Thanks for your solution. @emilioalvap
I'll update the result.