Elastic agent indices - ILM

Elastic Stack Elastic Agent
1

Hi All,

Currently using ELK stack 8.91.
Fleet enable. Elasticc-agent deployed on around 100 Servers/vm.
I notice that indexes will never be cleared. Seems to be a default behavior.
I need to know how to setup an Index Lifecycle Policies to clean old logs.
Can you let me know how can I setup this ?

Thanks in advance,
Regards,

Tyty

2

Hello and welcome,

You have two options in this case.

One is to simple edit the default lifecycle policy named logs to add a delete phase.

The second option is to create a custom template with a custom ILM for each dataset of each integration, which can be a lot of work.

To create a custom template you need to follow this documentation.

3

Hi leandrojmp,

Thank you for your reply.
I've just edit the default lifecycle policy.
kee you infomr on this.

Regards,
Tyty

4

Hi,

So, I change ILM logs as follow :
Maximum primary shard size : 10Mb
Maximum age : 2 days
Delete phase Move data into phase when: 12 hours

As far as I can see, used space is still growing. Data seems to be deleted but used space is a mess.
Any ideas ?
Regards,

5

You should increase the maxium primary shard size, 10 MB is way too small and not recommended at all. The recommended size for primary shard is something close to 50 GB.

With a primary shard size of 10 MB you risk creating too many indices/shards that can impact your cluster and even block writes if you reach the maximum shards allowed.

Are the backing indices for the Elastic Agent data streams being deleted? If they are being deleted, then I'm not sure what is the issue.

Depending on the integrations you are using Elastic Agente can be very noisy.

What is the space available in your cluster?

6

So I restore my snaphot VM and adjust my config as follow :
Maximum primary shard size : 50GB
Maximum age : 2 days
Delete phase Move data into phase when: 12 hours

Are the backing indices for the Elastic Agent data streams being deleted?
How can i know this information ?

Space available actually is around 100gb. If needed I can add more, but I want to control space used before.

Anyway, thanks for your answers.

regards,

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.