# How to limit the storage size of my indeces and automatically generate new index after reaching the assign storage size?

**URL:** <https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092>\
**Category:** Kibana\
**Tags:** ilm-index-lifecycle-management\
**Created:** [April 24, 2022, 9:44am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092 "2022-04-24T09:44:17Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [April 24, 2022, 9:44am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/1 "2022-04-24T09:44:17Z")

</div>

Good Day ELK masters, just want to ask on how can i limit my indeces/day storage size into i.e. 2gb? Please can somene help me i already created index life cycle policy but still it doesn't generate new index when exceed by 2gb

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/8/581d6100a24da934ed9703bef668ae552ae488ca.jpeg)

---

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [April 24, 2022, 10:39am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/2 "2022-04-24T10:39:37Z")

</div>

please can someone help me. TIA

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 24, 2022, 2:54pm UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/3 "2022-04-24T14:54:12Z")

</div>

Hi @renatoa12

First, please be patient. This is a community forum. Please do not ping multiple times, especially after only 1 hour In my experience, sometimes that'll actually have the opposite effect on getting help. There are many questions here and yours is no more important than any other topic.

Can you provide the actual ILM policy you defined? We can't help without seeing it.

Second, it seems like the index is still less than 2 GB, so I'm unclear why you're concerned it's not working?

Please post the ILM policy and perhaps we can help.

Go to Kibana- Dev Tools

And use this API and show us what your ILM policy is

> **[Get lifecycle policy API | Elasticsearch Guide \[8.1\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-get-lifecycle.html)**

Also as a note ILM is a background process and the data sizes may not be exact .. especially with small sizes such as 1 or 2 GB ... I would think +/- 5% or so... as indices / setting larger that percentage is small.. NOTE this is just an estimate from experience.

Here are some thoughts on the topic ...I posted a bit about it [here](https://discuss.elastic.co/t/rollover-goes-beyond-the-condition/207442/2) and another elastician [here](https://discuss.elastic.co/t/lifecycle-policies-dont-work/276218/13)

---

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [April 29, 2022, 2:46am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/4 "2022-04-29T02:46:50Z")

</div>

HI @stephenb ,

Sorry sir for being impatient 🙂 , btw for the sake of testing i tried to set the maximum storage size into 1mb but it still increasing and not creating/generating another index after reaching the assign maximum storage size.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 29, 2022, 2:49am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/5 "2022-04-29T02:49:06Z")

</div>

Yeah 1mb won't work... That is what I am trying to tell you...

We see this over and over the people try to test which such tiny values that's not what ILMs for.

I linked to some post explaining.

Also, you haven't actually posted your ILM or your index template that shows which ILM policy is being used So we really can't debug it unless you actually provide the actual configurations.

---

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [April 29, 2022, 3:29am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/6 "2022-04-29T03:29:33Z")

</div>

Hi sir @stephenb ,

Thanks for the reply, noted sir does it work if i change the maximum storage size into 1gb?? Btw i provided screenshots for my settings

Index Lifecycle Policies:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/2/920712c2eae624ffa62a51e1a1659d183e087d58.png)

Index Template:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/3/e3b4344de876f13ff67aa0d5bbd68056c972f359.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8eb5446f5b5a5627de3d23227e1bb4fb282621b0.png)

Index Management:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0cb539a928843a753027b7687839673fb0e2e014.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/d/8dc8c980a07285ceda222915d1425b67da455d50.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 29, 2022, 5:16am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/7 "2022-04-29T05:16:56Z")

</div>

What size did it rollover at?

---

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [May 8, 2022, 7:27am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/8 "2022-05-08T07:27:09Z")

</div>

HI @stephenb ,

There is no specific size it rollover per day it even reach at the size of 80gb+ thats why i want to limit the maximum storage size into 50gb only but still it doesnt work

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 8, 2022, 2:40pm UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/9 "2022-05-08T14:40:05Z")

</div>

I am still confused... your index size you show in the image is 1.09GB that is but you say it is greater that 80GB? does not make sense to me but I will let that go.. .

Please provide output section of your logstash pipeline.... in **formatted text** not a screen shot

Also did you bootstrap the initial index with the write alias? [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-bootstrap) ? I suspect not .. meaning you are not writing to the write alias which means the index will never roll over.

```auto
PUT big-ip-waf-logs-2022.05.08-000001
{
  "aliases": {
    "logstash": {
      "is_write_index": true
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [May 10, 2022, 6:56am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/10 "2022-05-10T06:56:08Z")

</div>

Hi @stephenb,

Thanks for the reply!  
Please see below how i configure my logstash.conf, as you can see my index name is "big\_ip\_waf\_logs-%{+YYY.MM.dd}-000001" this is to automatically generate the name same with the name i provide when running the write alias code.

**Note: i only set maximum storage size into 400kb just for the testing purposes on my lab.**

Logstash.conf:

```auto
input {
  udp {
    port => 514
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
   mutate {
                gsub => ["message","\"",""]
    }
    csv {
                separator => "#"
        columns => [
                        "header",
                        "geo_location",
                        "ip_address_intelligence",
                        "src_port",
                        "dest_ip",
                        "dest_port",
                        "protocol",
                        "method",
                        "uri",
                        "x_forwarded_for_header_value",
                        "request_status",
                        "support_id",
                        "session_id",
                        "username",
                        "violations",
                        "violation_rating",
                        "attack_type",
                        "query_string",
                        "policy_name",
                        "sig_ids",
                        "sig_names",
                        "sig_set_names",
                        "severity",
                        "request",
                        "violation_details"

                ]
    }
    grok {
       match => { "header" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} ASM:%{IP:source_ip}" }
    }
    mutate {
                remove_field => ["message", "header"]
    }
    mutate {
                gsub => ["sig_set_names", "},{", "}#{"]
    }
date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}

output {
  elasticsearch { 
	hosts => ["localhost:9200"] 
	index => "big_ip_waf_logs-%{+YYY.MM.dd}-000001"
}
}

```

I also created write alias as you mention. take note that i run this first before generating indices so that it will create index first with the same format of my index configure in logstash.conf.

```auto
PUT big_ip_waf_logs-2022.05.08-000001
{
  "aliases": {
    "logstash": {
      "is_write_index": true
    }
  }
}

```

It generated/rollover when tried to execute retry lifecycle policy. But i notice that it still continue to add the data into my first index which is "big\_ip\_waf\_logs-2022.05.10-000001" not in my"...-000002" index.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/e/4efd5e1b23b344c2b88e9335ebbab6b8cc85cce3.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c93d3502af1f4a9ae4e2330cf07376d98da595b2.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/c/ec128ba761c5942d4b5e1cce0997bb264a8bb96d.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 10, 2022, 12:48pm UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/11 "2022-05-10T12:48:44Z")

</div>

So the logstash output should be writing to the write alias not the actual index.

Cleanup and try again

```auto
output {
  elasticsearch { 
	hosts => ["localhost:9200"] 
	index => "logstash" <! ----THIS NEEDS TO BE ThE WRITE ALIAS not the concrete / actual index
}

```

and as a reminder again 400KB will not really work ... but I think I get it...

Start the process with the write alias above then force the rolloover I think it will work the way you want.

---

<div class="post-metadata">

**Author:** ![renatoa12](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@renatoa12](https://discuss.elastic.co/u/renatoa12)\
**Post date:** [May 11, 2022, 9:39am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/12 "2022-05-11T09:39:37Z")

</div>

Hi @stephenb,

Thank you very much sir, its now generated the 2nd index 🙂 i just change the index into write alias as you mention in your previous comment, also i change the name of my writealias match with my index name to avoid issue 🙂

```auto
PUT logstash-000001 //name should start with your index name(logstash) in logstash.conf file
{
  "aliases": {
    "logstash": {
      "is_write_index": true
    }
  }
}

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/226abda823a1882d3f3ec5b2ea36d1c45705bd45.png)

God bless you sir! hope you will continue helping newbies like me 🙂  
Thanks  
Renato

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2022, 9:39am UTC](https://discuss.elastic.co/t/how-to-limit-the-storage-size-of-my-indeces-and-automatically-generate-new-index-after-reaching-the-assign-storage-size/303092/13 "2022-06-08T09:39:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.