# Break up large index into multiple smaller equally sized indexes? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 1, 2021, 3:25am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745 "2021-03-01T03:25:16Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 1, 2021, 3:25am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/1 "2021-03-01T03:25:16Z")
Whats the best way to break up large index into multiple smaller equally sized indexes? I've looked at using reindex with a query but it seems extremely slow: ```auto XPOST localhost:9200/_reindex { "source": { "index": "$INDEX", "query": { "range": { "@timestamp": { "format": "date_time", "gte": "2021-01-24T00:00:00.000Z", "lt": "2021-01-24T13:00:00.000Z" } } } }, "dest": { "index": "$NEWINDEX" } } ``` I just want to break the index up into smaller chunks, doesn't have to be based on date, could be on size, say 5 equal pieces. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2021, 3:33am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/2 "2021-03-01T03:33:32Z")
You could use a new [ILM][ILM: Manage the index lifecycle | Elasticsearch Reference [7.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)) policy and then reindex into that. ---
**Author:** ![Eduardo\_Iglesias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eduardo_iglesias/32/78395_2.png) [@Eduardo\_Iglesias](https://discuss.elastic.co/u/Eduardo_Iglesias)\ **Post date:** [March 1, 2021, 4:42pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/3 "2021-03-01T16:42:35Z")
Hello @stevedwray , an option to tackle the slowness while reindexing is to disable the replicas of the new index and to throttle the requests. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 1, 2021, 7:11pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/4 "2021-03-01T19:11:33Z")
I have looked at ILM and do use it. However, this is a one-off where theres a historical index that I need to split up. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 1, 2021, 7:13pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/5 "2021-03-01T19:13:40Z")
I do that already. I noticed that when I am running a reindex with the above settings it takes an extremely long time before any documents are written. I'm thinking that its slowly working its way through the large index until it finds documents that match and then reindexes with those. As I'd need to repeat this operation several times, it would take far too long. So I'm wondering if theres an 'official' way to break up indexes into smaller indexes. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 2, 2021, 2:00am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/6 "2021-03-02T02:00:36Z")
You can still use it for that. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 2, 2021, 2:00am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/7 "2021-03-02T02:00:47Z")
> [@stevedwray](#): > > So I'm wondering if theres an 'official' way to break up indexes into smaller indexes. Nope, you're free to choose your own approach ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 2, 2021, 2:02am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/8 "2021-03-02T02:02:30Z")
Let me rephrase that; "A best-practise way to break up indexes into smaller indexes". ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 2, 2021, 2:09am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/9 "2021-03-02T02:09:45Z")
The answer there is still the same 🙂 ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 2, 2021, 2:11am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/10 "2021-03-02T02:11:53Z")
So the best way to do it is using ILM for this one-off job? Is that how you'd approach it? And there is no 'best practice', just people do it however they figure it out? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 2, 2021, 2:22am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/11 "2021-03-02T02:22:45Z")
That's how I would do it, yes. It allows you to define and automatically create the indices based on the size you want. It also makes sure they are easily queryable via an alias. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 2, 2021, 2:23am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/12 "2021-03-02T02:23:43Z")
I'm trying it out, created an ILM and attached it to this index. I'll see how it turns out in the morning! Thanks for the tip. It does seem a bit nicer than running lots of curl API calls with queries etc. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 2, 2021, 7:32am UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/13 "2021-03-02T07:32:42Z")
What is the reason for breaking it up? If the shards are too large and affecting performance you can simply use the [split index API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-split-index.html) to increase the number of primary shards. Querying a single index with X shards is not much different to querying X indices with 1 primary shard each from a performance perspective. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 3, 2021, 7:38pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/14 "2021-03-03T19:38:41Z")
The index contains packetbeat data and was allowed to grow too large without ILM rotating it. A lot of that data is flow which is not needed. The index covers a period thats got valuable historical data. I've been trying to re-index it into another index without the flow data but it goes incredibly slowly, as in 4 days later it hasn't done a quarter. I've done the same with other indexes that were about 100G and they went very fast, hours. I reasoned that if I first break this large index down into indexes of 100G or so each, it could go faster. Divide and conquer. I have limited time to get this done before I move on to another job and I'd like to get it done before then. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 3, 2021, 7:50pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/15 "2021-03-03T19:50:33Z")
I made an ILM with the parameters set to hot phase, roll over on maximum index size of 50G and maximum age of 1 day and attached this index to it, this seems to be set fine, the index shows as being managed by this ILM. But it doesn't appear to have kicked in and done anything with the index. Do I need to manually fire it off. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2021, 10:24pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/16 "2021-03-03T22:24:37Z")
Manually fire what off? The reindex - yes. The ILM policy just handles index rotation, not the reindex. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 3, 2021, 10:40pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/17 "2021-03-03T22:40:42Z")
So if I've attached the ILM to an index, and told it to rotate the index when it exceeds 50GB, should that break it down into indexes of 50GB each? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2021, 10:41pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/18 "2021-03-03T22:41:09Z")
Yep. ---
**Author:** ![stevedwray](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@stevedwray](https://discuss.elastic.co/u/stevedwray)\ **Post date:** [March 3, 2021, 10:43pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/19 "2021-03-03T22:43:10Z")
It hasn't done that so I must have got something wrong. When I run `GET packetbeat-7.10.0-2021.01.20-000099/_ilm/explain` I get: ```auto { "indices" : { "packetbeat-7.10.0-2021.01.20-000099" : { "index" : "packetbeat-7.10.0-2021.01.20-000099", "managed" : true, "policy" : "split_large_index", "lifecycle_date_millis" : 1611716731954, "age" : "35.81d", "phase" : "warm", "phase_time_millis" : 1611716735321, "action" : "complete", "action_time_millis" : 1611716735493, "step" : "complete", "step_time_millis" : 1611716735493, "phase_execution" : { "policy" : "split_large_index", "phase_definition" : { "min_age" : "0d", "actions" : { } }, "version" : 3, "modified_date_in_millis" : 1614800913433 } } } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2021, 10:43pm UTC](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745/20 "2021-03-03T22:43:46Z")
Can you share the policy and how you attached it to the index? [Next page](https://discuss.elastic.co/t/break-up-large-index-into-multiple-smaller-equally-sized-indexes/265745.md?page=2)