Deleting index after a period but keeping part of it

[kmohd]

Hello everyone, is there a way to delete the whole index to clear space after 10 days, however, i would like to keep a part of the index like for example in an allow/deny field from FW logs, is there a way to keep those two fields and delete the rest of the index?

[Juanma (Juan Manuel Nieto)]

Hello kmohd

I think one way to do to that would be reindex before delete as follows:

POST _reindex
{
  "source": {
    "index": "index_to_delete",
    "_source": ["field_to_keep1", "field_to_keep2.."]
  },
  "dest": {
    "index": "index_to_keep"
  }
}

With this you can keep a copy of the index with the fields that you want and then remove the desired one, if do it please remember to check the data in the new index before delete it

[kmohd]

Hello Juanma, thank you for your prompt response and assistance can you please explain further how to automate this? I am fairly new to ELK.

[Juanma (Juan Manuel Nieto)]

Hello Kmohd

One of the ways to automate this its a shell script with cron.
But before that I forgot something important, is this index beeying indexed while you want to delete it?

if for example it isn't and follows daily pattern for example firewall_log-2019-06-30 and a given fields "action" and "event.date" a the "easy way" would be something like this:

#!/bin/bash

# Obtaining date of 10 days ago with index name format.
DATE_FORMATTED=date +%F --date "10 days ago"

# Reindex with desired fields
curl -XPOST http://es_server:9200/_reindex -d "
{
  \"source\": {
      \"index\": \"firewall_log-${DATE_FORMATTED}\",
      \"_source\":[\"action\",\"event.date\"]
  },
  \"dest\":{
    \"index\":\"firewall_log-${DATE_FORMATTED}-reduced\"
  }
}"

# Delete de index
curl -XDELETE http://es_server:9200/firewall_log-${DATE_FORMATTED}

You can do this without the date command using date math, but its a bit more tricky just because you have to use URI encoded special characters https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html

I also left you here the documentation of the _reindex API https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

You also can use elastic curator with cron instead of shell scripts in order to manage your indices https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

Its important to remember that the given example can only be used the the index is not having any indexing load on the contrary you will lose data.

The script given is very basic, I would implement some controls etc.. before use it with important data.

Please remember to test everything before use it with important data

I hope this help

[kmohd]

Thank you, that helps alot, yes it is being indexed but my main question was, I was using an Index Lifecycle Policy to delete the index within 10 days, i wanted to include in the policy itself to keep some fields, dont know if this is possible.. but your solution is also valid for me.

[Juanma (Juan Manuel Nieto)]

Hello @kmohd

I dont currently use ILM but after checking the documentation it doesn't seems to be possible.

The only way (as far as I know) that could be done is with reindex, and checking ILM policies actions https://www.elastic.co/guide/en/elasticsearch/reference/7.2/_actions.html it seems not to be possible.

[dakrone (Lee Hinman)]

There is an open issue for this: https://github.com/elastic/elasticsearch/issues/42784 which explains some of the background. As Juan said, it's not currently possible, but may be in a future version of ILM.