# Curator - Reindex does not complete on first run **URL:** \ **Category:** Elasticsearch\ **Created:** [June 5, 2018, 1:15pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629 "2018-06-05T13:15:32Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![barryp](https://avatars.discourse-cdn.com/v4/letter/b/7ab992/32.png) [@barryp](https://discuss.elastic.co/u/barryp)\ **Post date:** [June 5, 2018, 1:15pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/1 "2018-06-05T13:15:33Z")
When run a reindex action in curator 5.5 for the first time, it doesn't reindex all docs. Only 998 docs. If I then run it for a second time it finishes the rest of the 870k docs. Logs do not show an error. reindex.yml: ``` --- actions: 1: action: create_index description: 'oinblogs: Create Weekly index.' options: disable_action: False continue_if_exception: True name: '' extra_settings: settings: number_of_shards: 6 number_of_replicas: 1 2: description: 'oinblogs: Reindex oinblogs- daily indices into week index after x day' action: reindex options: disable_action: False wait_interval: 30 max_wait: -1 request_body: source: index: REINDEX_SELECTION dest: index: '' filters: - filtertype: pattern kind: prefix value: oinb- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 16 ``` Did some else encounter this problem? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [June 5, 2018, 1:39pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/2 "2018-06-05T13:39:06Z")
You should try using [`slices`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/option_slices.html), which uses the [sliced scroll](http://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-request-scroll.html#sliced-scroll) functionality in Elasticsearch to more effectively parallelize. You're probably hitting a hot-thread with your single threaded approach. ---
**Author:** ![barryp](https://avatars.discourse-cdn.com/v4/letter/b/7ab992/32.png) [@barryp](https://discuss.elastic.co/u/barryp)\ **Post date:** [June 5, 2018, 1:58pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/3 "2018-06-05T13:58:35Z")
When using slices it continues untill 2.3k docs and then stops. Also a second time run doesn't work. The strange thing is that when **not** creating the index with the create\_index action, it's created by the reindex action and doesn't stop. 🤔 ---
**Author:** ![barryp](https://avatars.discourse-cdn.com/v4/letter/b/7ab992/32.png) [@barryp](https://discuss.elastic.co/u/barryp)\ **Post date:** [June 5, 2018, 2:13pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/4 "2018-06-05T14:13:30Z")
It looks like it has something to do with the number of shards. 6 and 7 shards per week index do not work. 5 works like a sharm. I have a 3 node cluster 16 gb per elasticsearch server Daily index: Total: 24.4 GB Primaries: 12.2 GB Documents: 32.5m I want to put them into a weekly index after 1 week, to reduce the number of open indices/shards. We want to keep 1,5 months of data. Is there a max number of shards per index? Is it necessary to reindex the daily into weekly? What is the adviced number of shards for the day index and for the week index? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [June 5, 2018, 2:18pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/5 "2018-06-05T14:18:35Z")
Oh! It's probably a timing issue with the cluster state. You don't need to pre-create the index. The reindex will do that for you. Additionally, there is absolutely no reason to have a 6 or 7 shard index on a 3 node cluster. You should not use more shards than data nodes, generally speaking. In fact, best practices is to not use more shards than is necessary at all. You can easily fit 5GB to 50GB in a single shard, so multiple shards is overkill. If you're reindexing _down_ the number of shards, you should probably go to 1 shard per index. Have you considered using [`shrink`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/shrink.html) instead of reindexing down? ---
**Author:** ![barryp](https://avatars.discourse-cdn.com/v4/letter/b/7ab992/32.png) [@barryp](https://discuss.elastic.co/u/barryp)\ **Post date:** [June 5, 2018, 2:30pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/6 "2018-06-05T14:30:06Z")
If I don't pre-create the index, it will create an index having 5 shards. Which is not evenly spread over the cluster nodes. I though this was adviced. My week indices will grow to: Total: 7 \* 25 = 175 GB Primairies: 7 \* 12 = 84 GB So I could configure my weekly index to have 3 P shards. Should I configure my daily index to have 1 P shard and 2 replica's? I not per se trying to reduce the number of indices/shards. I'm trying to configure it in the best way. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [June 5, 2018, 2:43pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/7 "2018-06-05T14:43:34Z")
How is 7 balanced? You should use index templates to set the number of shards for new indices, as well as setting the appropriate mappings. ---
**Author:** ![barryp](https://avatars.discourse-cdn.com/v4/letter/b/7ab992/32.png) [@barryp](https://discuss.elastic.co/u/barryp)\ **Post date:** [June 5, 2018, 2:49pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/8 "2018-06-05T14:49:53Z")
Thanks for the advice, I will also look into Shrink. ---
**Author:** ![barryp](https://avatars.discourse-cdn.com/v4/letter/b/7ab992/32.png) [@barryp](https://discuss.elastic.co/u/barryp)\ **Post date:** [June 5, 2018, 2:50pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/9 "2018-06-05T14:50:30Z")
> [@barryp](#): > > My week indices will grow to: > Total: 7 \* 25 = 175 GB > Primairies: 7 \* 12 = 84 GB 7 as in 7 days in a week. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [June 5, 2018, 4:36pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/10 "2018-06-05T16:36:53Z")
> [@barryp](#): > > It looks like it has something to do with the number of shards. > 6 and 7 shards per week index do not work. This is why I asked about the number 7. You still wouldn’t need a shard per day. And don’t count the total, only the amount per shard, like with your primaries. If you have 12.2G per day, you could go 4 days in a single shard. Your weekly index should only need 2 shards, to keep below the 50G/shard recommendation. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 3, 2018, 4:36pm UTC](https://discuss.elastic.co/t/curator-reindex-does-not-complete-on-first-run/134629/11 "2018-07-03T16:36:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.