How to specify ILM policies in Elastic agent policy config?

Elastic Stack Elastic Agent
1

I have about 2000 Elastic agents (version 8.9.0) connected to a system with 3 Fleet servers (version 8.9.0).

We have about 20 different agent policies, because the various Elastic agents are sending
slightly different logs, and for certain cases we need to specify specific pipelines to process the logs.

In the Fleet UI, if I go to kbn:/app/fleet/data-streams ,
I can see each dataset associated with each Elastic Agent.

Is it possible to specify ILM policies for each elastic agent's data set in the agent policy?

Or do ILM policies for an elastic agent's data set need to be specified outside of the agent policy?

I'm having difficulty figuring this out.
Thanks for any help.

2

Elastic Agent uses the same ILM policy for everything, it is an ILM policy named logs, every integration will use this same ILM policy.

You can customize the data retention using a custom ILM policy according to the documentation, but this is a manual process that needs to be done for every data set in every integration.

For example, if an integration have 10 data sets, you will need to create 10 custom templates, you can use the same ILM policy for these templates, but you will need one template per dataset per integration.

The same thing applies to custom ingest pipelines, you can create custom ingest pipelines per dataset per integration.

3

Thank you @leandrojmp . Your response is very concise and accurate.

I have a related post about elastic agent datasets: Performance impact of setting 'namespace' in Elastic agent policy config

If I have 2000 Elastic agents, 20 agent policies, does that mean that:

  1. Look up the Elastic agents that I want to have a customized data retention policy
  2. For each Elastic agent, I need to look up the datastream used by that agent going to kbn:/app/fleet/data-streams
  3. For each datastream that I want to have a customized ILM, I would need to use custom ILM policies according to the documentation

Is my understanding correct?

4

Also, you mentioned that the default ILM policy used by Elastic agent is logs.

In Elastic 8.9.0, if I navigate to:

kbn:/app/management/data/index_lifecycle_management/policies

I only see these ILM policies listed:

  • .items-default
  • .lists-default
  • .monitoring-8-ilm-policy
  • .preview.alerts-security.alerts-policy
  • Systems-Security-Policy
  • filebeat
  • heartbeat
  • kibana-event-log-policy
  • kibana-reporting
  • log-explorer-policy
  • metricbeat
  • my-data-lifecycle

Do you know which ILM policy is being used by datastreams created by Elastic agent?

5

I'm not using 8.9, but unless anything has changed the policy is still named logs, you need to toggle the option Include managed system policies to show the managed policies.

You need to follow the steps in the documentation, which is basically what you described, but yes, you will need a template for every data stream.

6

Oh OK! I did the following:

  1. navigated to kbn:/app/management/data/index_lifecycle_management/policies
  2. clicked on Include managed system policies

At that point I could see the logs ILM policy.

Thank you for your clear and concise explanations!

They really helped me out!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.