# Handling high task count on daily index rollover **URL:** \ **Category:** Elasticsearch\ **Created:** [January 5, 2021, 1:42pm UTC](https://discuss.elastic.co/t/handling-high-task-count-on-daily-index-rollover/260214 "2021-01-05T13:42:19Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![OranShuster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oranshuster/32/81779_2.png) [@OranShuster](https://discuss.elastic.co/u/OranShuster)\ **Post date:** [January 5, 2021, 1:42pm UTC](https://discuss.elastic.co/t/handling-high-task-count-on-daily-index-rollover/260214/1 "2021-01-05T13:42:19Z")
Some stats about the cluster Version 6.5.0 3 master nodes 12 hot data+ingest nodes 16 vCPU 32GB mem (16GB heap) each 12 warm data+ingest nodes 8 vCPU 32GB mem (16GB heap) each 1 ingest node (added recently for testing) Ingestion is done with a ALB that routes to all the non-master nodes (warm+hot+client) data is sent from a flink job using the java ES client + filebeat pods indices have no static mappings and dynamic mapping is used Some indices have pretty complex structure but nothing too nested or exotic Hot nodes hold data 3-4 days back and warm nodes hold all the rest of the data (depends on index that can be 30 days back) Average number of shards per hot node is 40 with total size of 1.3TB (1.5B docs) Since we reduces the amount of shards per index significantly we have no issues during the day ingesting, indexing and querying data. However, every day at 00:00 when our filebeat pods write to the new indexes, the task queue completely fills up. It takes about 5 hours to stop seeing rejections completely and i know (but did not measure) that it takes some time for the new indices to show up How can this be improved? i guess that upgrading to a new version can help and we are planning to do so but we want to find a solution in the meantime. I believe the amount of index/shards per node is completely within recommended limits. is it a CPU intensive operation? memory intensive? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 5, 2021, 8:29pm UTC](https://discuss.elastic.co/t/handling-high-task-count-on-daily-index-rollover/260214/2 "2021-01-05T20:29:06Z")
I relatively simple way to get around this would be to switch to the rollover API so that indices roll over based on time and/ size. This will make the various indices likely to roll over at different times which would spread out the work. For any other suggestions it would help to know how many daily indices/shards you are indexing into and how many shards and indices you have in the cluster as a whole as this drives the size of the cluster state. ---
**Author:** ![OranShuster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oranshuster/32/81779_2.png) [@OranShuster](https://discuss.elastic.co/u/OranShuster)\ **Post date:** [January 5, 2021, 9:11pm UTC](https://discuss.elastic.co/t/handling-high-task-count-on-daily-index-rollover/260214/3 "2021-01-05T21:11:51Z")
1. Lets say today is 1/5. If we create the indices for 1/6 during the day, it will still need to create the dynamic mapping which is what i would guess the more time consuming task, isn't it? 2. We are still considering using the size rollover option. 3. The daily stats are, on average: indices - 38 shards - 160 (we aim for about 50GB per shard) doc count - 4.7B storage size - 4.6TB (1 replica for each index) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 5, 2021, 9:21pm UTC](https://discuss.elastic.co/t/handling-high-task-count-on-daily-index-rollover/260214/4 "2021-01-05T21:21:19Z")
> [@OranShuster](#): > > it will still need to create the dynamic mapping which is what i would guess the more time consuming task, isn't it? That'd be my guess too. Dynamic mappings are useful for getting started but they're expensive since they involve the master in indexing which can be a bottleneck as there's only one master. The simple solution is to avoid them and define your mappings up-front as far as possible. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 2, 2021, 9:21pm UTC](https://discuss.elastic.co/t/handling-high-task-count-on-daily-index-rollover/260214/5 "2021-02-02T21:21:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.