# Deleting index after a period but keeping part of it

**URL:** <https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [June 30, 2019, 9:19am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183 "2019-06-30T09:19:00Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmohd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmohd/32/49090_2.png) [@kmohd](https://discuss.elastic.co/u/kmohd)\
**Post date:** [June 30, 2019, 9:19am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/1 "2019-06-30T09:19:00Z")

</div>

Hello everyone, is there a way to delete the whole index to clear space after 10 days, however, i would like to keep a part of the index like for example in an allow/deny field from FW logs, is there a way to keep those two fields and delete the rest of the index?

---

<div class="post-metadata">

**Author:** ![Juanma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juanma/32/49076_2.png) [@Juanma](https://discuss.elastic.co/u/Juanma)\
**Post date:** [June 30, 2019, 9:28am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/2 "2019-06-30T09:28:50Z")

</div>

Hello kmohd

I think one way to do to that would be reindex before delete as follows:

```
POST _reindex
{
  "source": {
    "index": "index_to_delete",
    "_source": ["field_to_keep1", "field_to_keep2.."]
  },
  "dest": {
    "index": "index_to_keep"
  }
}

```

With this you can keep a copy of the index with the fields that you want and then remove the desired one, if do it please remember to check the data in the new index before delete it 🙂

---

<div class="post-metadata">

**Author:** ![kmohd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmohd/32/49090_2.png) [@kmohd](https://discuss.elastic.co/u/kmohd)\
**Post date:** [June 30, 2019, 9:43am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/3 "2019-06-30T09:43:49Z")

</div>

Hello Juanma, thank you for your prompt response and assistance 🙂 can you please explain further how to automate this? I am fairly new to ELK.

---

<div class="post-metadata">

**Author:** ![Juanma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juanma/32/49076_2.png) [@Juanma](https://discuss.elastic.co/u/Juanma)\
**Post date:** [June 30, 2019, 10:45am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/4 "2019-06-30T10:45:20Z")

</div>

Hello Kmohd

One of the ways to automate this its a shell script with cron.  
But before that I forgot something important, is this index beeying indexed while you want to delete it?

if for example it isn't and follows daily pattern for example firewall\_log-2019-06-30 and a given fields "action" and "event.date" a the "easy way" would be something like this:

```
#!/bin/bash

# Obtaining date of 10 days ago with index name format.
DATE_FORMATTED=date +%F --date "10 days ago"

# Reindex with desired fields
curl -XPOST http://es_server:9200/_reindex -d "
{
  \"source\": {
      \"index\": \"firewall_log-${DATE_FORMATTED}\",
      \"_source\":[\"action\",\"event.date\"]
  },
  \"dest\":{
    \"index\":\"firewall_log-${DATE_FORMATTED}-reduced\"
  }
}"

# Delete de index
curl -XDELETE http://es_server:9200/firewall_log-${DATE_FORMATTED}

```

You can do this without the date command using date math, but its a bit more tricky just because you have to use URI encoded special characters [https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html)

I also left you here the documentation of the \_reindex API [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)

You also can use elastic curator with cron instead of shell scripts in order to manage your indices [https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html)

**Its important to remember that the given example can only be used the the index is not having any indexing load on the contrary you will lose data.**

**The script given is very basic, I would implement some controls etc.. before use it with important data.**

**Please remember to test everything before use it with important data**

I hope this help 🙂

---

<div class="post-metadata">

**Author:** ![kmohd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmohd/32/49090_2.png) [@kmohd](https://discuss.elastic.co/u/kmohd)\
**Post date:** [June 30, 2019, 10:58am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/5 "2019-06-30T10:58:50Z")

</div>

Thank you, that helps alot, yes it is being indexed but my main question was, I was using an Index Lifecycle Policy to delete the index within 10 days, i wanted to include in the policy itself to keep some fields, dont know if this is possible.. but your solution is also valid for me. 🙂

---

<div class="post-metadata">

**Author:** ![Juanma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juanma/32/49076_2.png) [@Juanma](https://discuss.elastic.co/u/Juanma)\
**Post date:** [June 30, 2019, 11:24am UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/6 "2019-06-30T11:24:45Z")

</div>

Hello @kmohd

I dont currently use ILM but after checking the documentation it doesn't seems to be possible.

The only way (as far as I know) that could be done is with reindex, and checking ILM policies actions [https://www.elastic.co/guide/en/elasticsearch/reference/7.2/\_actions.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/_actions.html) it seems not to be possible. ☹

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [July 1, 2019, 3:19pm UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/7 "2019-07-01T15:19:56Z")

</div>

There is an open issue for this: [https://github.com/elastic/elasticsearch/issues/42784](https://github.com/elastic/elasticsearch/issues/42784) which explains some of the background. As Juan said, it's not currently possible, but may be in a future version of ILM.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 29, 2019, 3:20pm UTC](https://discuss.elastic.co/t/deleting-index-after-a-period-but-keeping-part-of-it/188183/8 "2019-07-29T15:20:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.