Elastic Agent Logs Disk Usage and Shard Count

Elastic Stack Elasticsearch
1

I’m looking for some help on how to better handle log data generated by Elastic. For example, what looks to be Elastic Agent log data is consuming almost 4.5tb on our frozen tier, double the size of any other dataset on our cluster.

image

Looks like its using the builtin ‘logs’ ILM policy, which has a 25GB/5day rollover to frozen before being deleted at 365 days. This is also causing a shard capacity issue.

How do we better handle these indices?

Some of the indices that are causing issues:

.ds-logs-elastic_agent-default-

.ds-logs-elastic_agent.filebeat-default-

.ds-logs-elastic_agent.metricbeat-default-

.ds-logs-system.application-default-

.ds-logs-system.security-default-
2

Hi @jnpetty ... Welcome Back :slight_smile:

tl;dr; You have complete control over the Lifecycle of all those indices

Some appear to be agent related, others like the system one are just normal logs from integrations etc...

Pretty sure that is not the default of the built-in policy; I suspect that those policies have been edited at some point.... which is fine.

Curious What version are you on... and what version did you come from?

But in short, you can Create and Apply policy to any / all indices/data streams you like, using the custom ILM policies and applying them by the @custom templates (or directly editing the defaults, which is generally discouraged)

Let Us Know some specifics

3

Hi @stephenb, thanks for getting back to me. We are on 8.17 and I think thats the version we started on when we moved to Elastic Cloud. Before that we started our on prem instance on version 5 or 6… i think.

Our SE and I didnt make any changes to the default “logs” policy as far as I know because we started making custom policies that matched our on prem indices. But maybe we did. Whats the default settings for that policy?

I can work on creating custom policies, with hopes of getting them back under control. I can look at the index and see the assigned ILM policy, but I havent figured out how to identify what template that is associated with. I assume there is a way, right?

Also, is there any documentation on what data goes into these types of indexes? It would help me identify what retention policy is needed.

4

Hi @jnpetty

All the defaults for logs policies in 8.17

"Roll over when an index is 30 days old or any primary shard reaches 50 gigabytes."

Do a GET on the policy, I suspect it will not be version 1

GET _ilm/policy/logs

{
  "logs": {
    "version": 1, <<<<
...

The ILM Policy is defined in the Templates (or component template)

Run

GET _data_stream/logs-default-default
{
  "data_streams": [
    {
      "name": "logs-default-default",
      "timestamp_field": {
        "name": "@timestamp"
......
      ],
      "generation": 1,
      "_meta": {
        "managed": true,
        "description": "default logs template installed by x-pack"
      },
      "status": "YELLOW",
      "template": "logs", <<< HERE
      "ilm_policy": "logs", <<< HERE 
      "next_generation_managed_by": "Index Lifecycle Management",
.....
    }
  ]
}
5

Hi @stephenb You are correct, looks like its on version 3. So we must have made some changes at some point.

Thanks I was able to locate the template. I will get an updated policy created

image
image668×657 40.4 KB

Side question, whats the best way to move existing indices to these new policy? I can do it one-by-one within Index Management, but it wont let me bulk change.

I have tried using Dev Tools

PUT partial-.ds-logs-elastic_agent.filebeat-default-2025.10*/_settings
{"index.lifecycle.name": "new-test-policy"}

Put I get an error

image
image453×157 6.04 KB