# Is Rally a good choice to create data lake? **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [August 30, 2021, 1:59pm UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826 "2021-08-30T13:59:16Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Post date:** [August 30, 2021, 1:59pm UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/1 "2021-08-30T13:59:16Z")
hey there, regularly I have to create a clusters from scratch and test their performances when indices contain data at least from 20 days. The possibility to have a data lake will give me the chance to avoid to wait for that period. So I started to think about Rally usage. I tried to create a custom track from an already existing index on a test cluster. Operation was completed correctly but I would like to know if, from your point of view, this could be a good way to create a data lake. thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 30, 2021, 9:12pm UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/2 "2021-08-30T21:12:18Z")
What exactly do you mean by data lake here? ---
**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Post date:** [August 31, 2021, 6:20am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/3 "2021-08-31T06:20:05Z")
The chance to have every time I need a set of data (documents) to fill my new cluster. With a custom track saved wherever I want, I could load it into the cluster and speed up the operation since I am quite sure that Rally will be faster than real day-by-day ingest operation (i.e. Rally will take 1-2 days to bulk 20 days of documents). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 31, 2021, 6:24am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/4 "2021-08-31T06:24:25Z")
The by far fastest way to load data large volumes of data into a cluster for testing is by restoring a saved snapshot. This naturally means that the data will be covering a set time period rather than the last X days but that might be good enough for some types of performance testing, which you can perform using Rally. ---
**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Post date:** [August 31, 2021, 7:55am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/5 "2021-08-31T07:55:50Z")
thanks, this could be a good idea. But I have one doubt: Rally will compress (and then will uncompress, I know) the track, while snapshot will not do that. Is it a downside for snapshot approach? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 31, 2021, 7:59am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/6 "2021-08-31T07:59:05Z")
Indices are already compressed (although not quite as well as compressed JSON data) and you save a lot of time and resources as data does not need to be indexed. I have previously used this method to quickly restore terabytes of data to clusters for performance testing where indexing the full data set would have taken a lot longer. ---
**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Post date:** [August 31, 2021, 8:13am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/7 "2021-08-31T08:13:03Z")
Using Rally I have this scenario: 150 millions of docs 44 gb compressed 700 gb uncompressed (I suppose including 1 replica) Could I get a similar result also with snapshot? Apart from that, can you explain the sentence "save time as data does not need to be indexed"? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 31, 2021, 8:16am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/8 "2021-08-31T08:16:40Z")
Restoring a snapshot is a LOT faster and more efficient than indexing the same amount of data. If you want to build up even larger data volumes you can even restore the same snapshot multiple times if you rename the indices when you restore. You can naturally also combine the two methods and restore a background data set using snapshots and at gbe same time index new data using Rally. ---
**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Post date:** [August 31, 2021, 8:49am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/9 "2021-08-31T08:49:10Z")
ok, I will give a chance also to snapshot idea 😀 Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 28, 2021, 8:49am UTC](https://discuss.elastic.co/t/is-rally-a-good-choice-to-create-data-lake/282826/10 "2021-09-28T08:49:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.