# Transform and Aggregate Index with an Index Rate of 200k/s

**URL:** <https://discuss.elastic.co/t/transform-and-aggregate-index-with-an-index-rate-of-200k-s/288286>\
**Category:** Elasticsearch\
**Tags:** transforms, rollups\
**Created:** [November 3, 2021, 7:12am UTC](https://discuss.elastic.co/t/transform-and-aggregate-index-with-an-index-rate-of-200k-s/288286 "2021-11-03T07:12:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomx1](https://avatars.discourse-cdn.com/v4/letter/t/779978/32.png) [@tomx1](https://discuss.elastic.co/u/tomx1)\
**Post date:** [November 3, 2021, 7:12am UTC](https://discuss.elastic.co/t/transform-and-aggregate-index-with-an-index-rate-of-200k-s/288286/1 "2021-11-03T07:12:39Z")

</div>

We are thinking about gathering our DNS traffic via packetbeat and create statistics out of the collected data. Currently we have a query rate of about 200k/s and growing... Because of the tremendous amount of data, we would need to aggregate the data and push it to a smaller index.

But to me it seems that every possible solution which comes with Elasticsearch (transforms, rollups etc.) can't handle Indices with a very high index rate. For example, a transform with a frequency of 1s (lowest possible) and page size of 10k (default) I'm not able to process more than 10k documents per second which is way too low. Is there a way to achieve our goal?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [November 8, 2021, 4:39pm UTC](https://discuss.elastic.co/t/transform-and-aggregate-index-with-an-index-rate-of-200k-s/288286/2 "2021-11-08T16:39:23Z")

</div>

This is hard to answer, especially given the little details. Transform can go as fast as aggregations and indexing can go. Now `20x` is a huge gap between your POC implementation and your goal.

There is no fixed limit in transform or rollup, it is just a matter of resources. Despite the option to add more hardware I recommend our documentation for running [transform at scale](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-scale.html).

However, it would be nice if you can describe the use case in more detail. I wonder if transform is the right tool for your case, are you summarizing only on `terms`?

For in-depth questions and more detailed performance guidance it might be better to contact support.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 6, 2021, 4:39pm UTC](https://discuss.elastic.co/t/transform-and-aggregate-index-with-an-index-rate-of-200k-s/288286/3 "2021-12-06T16:39:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.