# How Rally will behave with Millions of operations in track.json?

**URL:** <https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [September 24, 2018, 8:06pm UTC](https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727 "2018-09-24T20:06:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![akhil-rane](https://avatars.discourse-cdn.com/v4/letter/a/ba9def/32.png) [@akhil-rane](https://discuss.elastic.co/u/akhil-rane)\
**Post date:** [September 24, 2018, 8:06pm UTC](https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727/1 "2018-09-24T20:06:06Z")

</div>

Hello all,

I have a use case wherein I am generating millions of queries and putting them into track.json as search operations. This will create a huge track.json file. Each elastic query itself is very huge containing a lot of filters. Will Rally be able to handle this?

Thanks,  
Akhil

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 24, 2018, 8:42pm UTC](https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727/2 "2018-09-24T20:42:36Z")

</div>

Generating an extremely large track.json file does not sound like a good idea, and I have a hard time seeing it working well. I would probably instead look into developing [a custom parameter source](https://esrally.readthedocs.io/en/stable/adding_tracks.html#custom-parameter-sources) that can read queries from one or more files and then use a search runner to execute them. Even though operations may not get individual names, you can add metadata to each query that allows you to analyse them in detail if you use an Elasticsearch instance as a [metrics store](https://esrally.readthedocs.io/en/stable/configuration.html?highlight=metrics%20store#advanced-configuration).

---

<div class="post-metadata">

**Author:** ![akhil-rane](https://avatars.discourse-cdn.com/v4/letter/a/ba9def/32.png) [@akhil-rane](https://discuss.elastic.co/u/akhil-rane)\
**Post date:** [September 28, 2018, 2:04pm UTC](https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727/3 "2018-09-28T14:04:47Z")

</div>

My queries file will be huge so I cannot load it into memory and then process it. Other way is reading it from disk line by line in params function. Don't you think this io operation will create bottleneck?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 28, 2018, 2:27pm UTC](https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727/4 "2018-09-28T14:27:13Z")

</div>

You should be able to read it line by line in a custom parameter source.

> [@akhil-rane](#):
>
> Don't you think this io operation will create bottleneck?

It might slow it down, but if it does not fit in memory, what are the options?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2018, 2:30pm UTC](https://discuss.elastic.co/t/how-rally-will-behave-with-millions-of-operations-in-track-json/149727/5 "2018-10-26T14:30:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.