I want to split the existing index to daily index

Elastic Stack Elasticsearch
#1

Hello,

I want to split the existing index of size 200gb into daily index, That index has data of 1 month.
I have setup of 8 core cpu and 16gb ram.
what query should I write on dev tools or is there any other way to execute this.
and how much time will it take to split the index.

I have tried this using logstash.
I took input as elasticsearch index and gave output according to the daily index.
but it takes a lot of time around 20 hours to split the 8 gb index after this the total size of daily indices increased more than 8 gb.

Please provide solution for this activity.
Kindly help.
Regards Rohit

#2

Hi Rohit,

I can't answer the first part, but you can use the rollover API to create daily a new daily index when conditions are triggered: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html

But you could try creating a new index, then using the reindex API on certain date ranges and so forth.

#3

Try using the _reindex API in Elasticsearch, with a timerange query to split things out.

#4

Thank you for your response,

I tried _reindex API Already but it also takes too much time to create index for this huge data

#5

Why are you looking to split the index? Is it due to query performance?

Which version of Elasticsearch are you using?

How many primary and replica shards does the index have?

#6

You can use _reindex with a painless script to change dynamically the index name based on the date of the event, something like this should work

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "source-index-name"
  },
  "dest": {
    "index": "destination-index-name"
  },
  "script": {
    "source": """
    
        def inputFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
        def myDate = inputFormat.parse(ctx._source['@timestamp']);
        
        def outputFormat = new SimpleDateFormat("yyyy-MM-dd");
        def outputDay = outputFormat.format(myDate);
        
        ctx._index = "destination-index-name-" + outputDay;
"""
  }
}
#7

You can also explore this processor inside an ingest pipeline when reindexing