# Track data on the load driver nodes

**URL:** <https://discuss.elastic.co/t/track-data-on-the-load-driver-nodes/154359>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [October 29, 2018, 12:34am UTC](https://discuss.elastic.co/t/track-data-on-the-load-driver-nodes/154359 "2018-10-29T00:34:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![akhil-rane](https://avatars.discourse-cdn.com/v4/letter/a/ba9def/32.png) [@akhil-rane](https://discuss.elastic.co/u/akhil-rane)\
**Post date:** [October 29, 2018, 12:34am UTC](https://discuss.elastic.co/t/track-data-on-the-load-driver-nodes/154359/1 "2018-10-29T00:34:22Z")

</div>

Hi all,

We have built a data generation tool that is distributed across 3 nodes. After data generation, we have 3 files containing documents. Let's say we have documents\_1.json, documents\_2.json, and documents\_3.json. Now suppose we have a distributed load test driver with one benchmark co-ordinator and 3 worker nodes. So do I need to copy 3 document files on all 3 worker nodes? If yes, what if document files are huge (in TBs). Do we need space required by 3 document files on all the worker nodes? Is there any workaround for this?

Thanks,  
Akhil

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [October 29, 2018, 7:06pm UTC](https://discuss.elastic.co/t/track-data-on-the-load-driver-nodes/154359/2 "2018-10-29T19:06:31Z")

</div>

Hello,

Yes the document corpora will occupy space on each load driver node.  
To avoid manually populating it on the load driver machines, you can store the (compressed) corpora in a network location accessible from your machines and use the `base-url` + `source_file` properties to specify the location, as per [the docs](https://esrally.readthedocs.io/en/stable/track.html?highlight=track%20definition). When you start Rally, it will download the corpora, if not present, and uncompress.

Having Rally "stream" the document corpora would introduce a potential bottleneck (would easily saturate the network interface, unless you have a dedicated network interface for the TCP/IP route towards the location of the docs as well as higher CPU+IO requirements) and thus taint the results.

If you are looking for a **manual way** to distribute large json files here are a few tips:

- such files compress very well. It's advisable to use something like `pbzip2 -v -k -m10000 documents.json` which takes advantage of all CPUs and saves time.
- distribute files to the right directories, while should be faster now that they are compressed.
- uncompress the files, again utilizing all CPUs with `pbzip2 -v -d -k -m10000 documents.json.bz2`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2018, 7:06pm UTC](https://discuss.elastic.co/t/track-data-on-the-load-driver-nodes/154359/3 "2018-11-26T19:06:32Z")

</div>

Closed on November 26, 2018, 7:06pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.