# Elastic Agent Logs Disk Usage and Shard Count

**URL:** <https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417>\
**Category:** Elasticsearch\
**Created:** [January 7, 2026, 4:28pm UTC](https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417 "2026-01-07T16:28:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [January 7, 2026, 4:28pm UTC](https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417/1 "2026-01-07T16:28:58Z")

</div>

I’m looking for some help on how to better handle log data generated by Elastic. For example, what looks to be Elastic Agent log data is consuming almost 4.5tb on our frozen tier, double the size of any other dataset on our cluster.

![image](https://us1.discourse-cdn.com/elastic/optimized/3X/4/6/4683ebd4820e1c4f62553306241b5273fe630284_2_690x40.png)

Looks like its using the builtin ‘logs’ ILM policy, which has a 25GB/5day rollover to frozen before being deleted at 365 days. This is also causing a shard capacity issue.

How do we better handle these indices?

Some of the indices that are causing issues:

```auto
.ds-logs-elastic_agent-default-

.ds-logs-elastic_agent.filebeat-default-

.ds-logs-elastic_agent.metricbeat-default-

.ds-logs-system.application-default-

.ds-logs-system.security-default-

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 7, 2026, 5:30pm UTC](https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417/2 "2026-01-07T17:30:59Z")

</div>

Hi @jnpetty ... Welcome Back 🙂

tl;dr; You have complete control over the Lifecycle of all those indices

Some appear to be agent related, others like the `system` one are just normal logs from integrations etc...

> **[Tutorials: Customize data retention policies | Elastic Docs](https://www.elastic.co/docs/reference/fleet/data-streams-ilm-tutorial)**
>
> These tutorials explain how to apply a custom ILM policy to an integration’s data stream. For certain features you’ll need to use a slightly different...

> [@jnpetty](#):
>
> Looks like its using the builtin ‘logs’ ILM policy, which has a 25GB/5day rollover to frozen before being deleted at 365 days. This is also causing a shard capacity issue.

Pretty sure that is not the default of the built-in policy; I suspect that those policies have been edited at some point.... which is fine.

Curious What version are you on... and what version did you come from?

But in short, you can Create and Apply policy to any / all indices/data streams you like, using the custom ILM policies and applying them by the `@custom` templates (or directly editing the defaults, which is generally discouraged)

Let Us Know some specifics

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [January 7, 2026, 6:48pm UTC](https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417/3 "2026-01-07T18:48:32Z")

</div>

Hi @stephenb, thanks for getting back to me. We are on 8.17 and I think thats the version we started on when we moved to Elastic Cloud. Before that we started our on prem instance on version 5 or 6… i think.

Our SE and I didnt make any changes to the default “logs” policy as far as I know because we started making custom policies that matched our on prem indices. But maybe we did. Whats the default settings for that policy?

I can work on creating custom policies, with hopes of getting them back under control. I can look at the index and see the assigned ILM policy, but I havent figured out how to identify what template that is associated with. I assume there is a way, right?

Also, is there any documentation on what data goes into these types of indexes? It would help me identify what retention policy is needed.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 7, 2026, 7:29pm UTC](https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417/4 "2026-01-07T19:29:27Z")

</div>

Hi @jnpetty

All the defaults for logs policies in 8.17

"Roll over when an index is 30 days old or any primary shard reaches 50 gigabytes."

Do a GET on the policy, I suspect it will not be version 1

```auto
GET _ilm/policy/logs

{
  "logs": {
    "version": 1, <<<<
...

```

> [@jnpetty](#):
>
> I can work on creating custom policies, with hopes of getting them back under control. I can look at the index and see the assigned ILM policy, but I havent figured out how to identify what template that is associated with. I assume there is a way, right?

The ILM Policy is defined in the Templates (or component template)

Run

```auto
GET _data_stream/logs-default-default
{
  "data_streams": [
    {
      "name": "logs-default-default",
      "timestamp_field": {
        "name": "@timestamp"
......
      ],
      "generation": 1,
      "_meta": {
        "managed": true,
        "description": "default logs template installed by x-pack"
      },
      "status": "YELLOW",
      "template": "logs", <<< HERE
      "ilm_policy": "logs", <<< HERE 
      "next_generation_managed_by": "Index Lifecycle Management",
.....
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [January 7, 2026, 9:44pm UTC](https://discuss.elastic.co/t/elastic-agent-logs-disk-usage-and-shard-count/384417/5 "2026-01-07T21:44:55Z")

</div>

Hi @stephenb You are correct, looks like its on version 3. So we must have made some changes at some point.

Thanks I was able to locate the template. I will get an updated policy created

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dcabbbbc4473a6ce3301e419d0da9392b60347e2.png)

Side question, whats the best way to move existing indices to these new policy? I can do it one-by-one within Index Management, but it wont let me bulk change.

I have tried using Dev Tools

```auto
PUT partial-.ds-logs-elastic_agent.filebeat-default-2025.10*/_settings
{"index.lifecycle.name": "new-test-policy"}

```

Put I get an error

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26f6197046f0c7d44d404b9d5fdc4ffe56199e80.png)