# Dec 7th, 2020: \[EN\] Using Rally as a data import/export tool **URL:** \ **Category:** Advent Calendar\ **Created:** [December 7, 2020, 9:16am UTC](https://discuss.elastic.co/t/dec-7th-2020-en-using-rally-as-a-data-import-export-tool/257820 "2020-12-07T09:16:23Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Laurent\_Huet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/laurent_huet/32/78677_2.png) [@Laurent\_Huet](https://discuss.elastic.co/u/Laurent_Huet)\ **Post date:** [December 7, 2020, 9:16am UTC](https://discuss.elastic.co/t/dec-7th-2020-en-using-rally-as-a-data-import-export-tool/257820/1 "2020-12-07T09:16:23Z")
[French version](https://discuss.elastic.co/t/256863) This post will show you how you can use [Rally](https://github.com/elastic/rally) to export data from one cluster to another. The idea is to extract the whole data from one or more indices in a (big) flat file and reuse it later to easily import in another cluster. Rally helps doing that very easily. Steps to follow: # Installing Rally Install Rally by following the [official documentation](https://esrally.readthedocs.io/en/stable/quickstart.html). This is optional if you are using Rally with Docker. You can test it's correctly working by running: ```auto ╰─$ esrally --version ``` # Exporting data Data export is made by [writing a scenario](https://esrally.readthedocs.io/en/stable/adding_tracks.html) with a command like: ``` ╰─$ esrally create-track \ --track=mytrack \ --target-hosts=URL.elastic-cloud.com:9243 \ --client-options="timeout:60,use_ssl:true,verify_certs:true,basic_auth_user:'USER',basic_auth_password:'PASSWORD'" \ --indices="kibana_sample_data_logs,kibana_sample_data_ecommerce,kibana_sample_data_flights" \ --output-path=~/Documents/Consulting/sandbox/rally ``` With docker, this command would be: ``` ╰─$ docker run -v ~/Documents/Consulting/sandbox/rally:/mnt \ elastic/rally create-track \ --track=mytrack-docker \ --target-hosts=URL.elastic-cloud.com:9243 \ --client-options="timeout:60,use_ssl:true,verify_certs:true,basic_auth_user:'USER',basic_auth_password:'PASSWORD'" \ --indices="kibana_sample_data_logs,kibana_sample_data_ecommerce,kibana_sample_data_flights" \ --output-path=/mnt ``` The following options are used: - `--track`: name your scenario - `--target-hosts`: where Rally should export data from (multiple servers are allowed) - `--indices`: which indices you would like to export - `--output-path`: where to export the data and the configuration - `--client-options`: client parameters, like authentication Once done, you can check what has been generated by running the following command: ``` ╰─$ esrally info --track-path=~/Documents/Consulting/sandbox/rally/mytrack ``` This will give you an idea of the whole size of the generated scenario: ```auto ________ / __\____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\ __,_/_/_/\__ , / / ____ / Showing details for track [mytrack]: * Description: Tracker-generated track for mytrack * Documents: 31,808 * Compressed Size: 2.1 MB * Uncompressed Size: 30.1 MB Schedule: ---------- 1. delete-index 2. create-index 3. cluster-health 4. bulk (8 clients) ------------------------------- [INFO] SUCCESS (took 0 seconds) ------------------------------- ``` You can compress all the files within one single archive and ship it to the infrastructure where your destination cluster is running. # Importing data In the following example, we are using a local development cluster. No need then for the `--client-options` parameter because our local cluster is not secured. ``` ╰─$ esrally --track-path=~/Documents/Consulting/sandbox/rally/mytrack --target-hosts=127.0.0.1:9200 --pipeline=benchmark-only ``` Once done, Rally will give some statistics about your cluster (ingestion rate for example): ```auto ________ / __\____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\ __,_/_/_/\__ , / / ____ / [INFO] Preparing file offset table for [/Users/laurent/Documents/Consulting/sandbox/rally/mytrack/kibana_sample_data_logs-documents.json] ... [OK] [INFO] Preparing file offset table for [/Users/laurent/Documents/Consulting/sandbox/rally/mytrack/kibana_sample_data_ecommerce-documents.json] ... [OK] [INFO] Preparing file offset table for [/Users/laurent/Documents/Consulting/sandbox/rally/mytrack/kibana_sample_data_flights-documents.json] ... [OK] [INFO] Racing on track [mytrack] and car ['external'] with version [7.9.2]. [WARNING] indexing_total_time is 10 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. Running delete-index [100% done] Running create-index [100% done] Running cluster-health [100% done] Running bulk [100% done] ------------------------------------------------------ _________ _____ / ____(_)___ _____/ / /___ / _________________ / /_ / / __\/__ `/ / \ __\/___ / __\/___ / _ \ / __/ / / / / / /_/ / /___ / / / __/ /_/ / / /__ / /_/ /_/_/ /_/\ __,_/_/ /____ /\ ___/\____ /_/ \___/ ------------------------------------------------------ | Metric | Task | Value | Unit | |---------------------------------------------------------------:|-------:|------------:|-------:| | Cumulative indexing time of primary shards | | 0.397 | min | | Min cumulative indexing time across primary shards | | 0.000166667 | min | | Median cumulative indexing time across primary shards | | 0.0681333 | min | | Max cumulative indexing time across primary shards | | 0.247217 | min | | Cumulative indexing throttle time of primary shards | | 0 | min | | Min cumulative indexing throttle time across primary shards | | 0 | min | | Median cumulative indexing throttle time across primary shards | | 0 | min | | Max cumulative indexing throttle time across primary shards | | 0 | min | | Cumulative merge time of primary shards | | 0.01685 | min | | Cumulative merge count of primary shards | | 1 | | | Min cumulative merge time across primary shards | | 0 | min | | Median cumulative merge time across primary shards | | 0 | min | | Max cumulative merge time across primary shards | | 0.01685 | min | | Cumulative merge throttle time of primary shards | | 0 | min | | Min cumulative merge throttle time across primary shards | | 0 | min | | Median cumulative merge throttle time across primary shards | | 0 | min | | Max cumulative merge throttle time across primary shards | | 0 | min | | Cumulative refresh time of primary shards | | 0.0601 | min | | Cumulative refresh count of primary shards | | 23 | | | Min cumulative refresh time across primary shards | | 0.000883333 | min | | Median cumulative refresh time across primary shards | | 0.00673333 | min | | Max cumulative refresh time across primary shards | | 0.0337333 | min | | Cumulative flush time of primary shards | | 0 | min | | Cumulative flush count of primary shards | | 0 | | | Min cumulative flush time across primary shards | | 0 | min | | Median cumulative flush time across primary shards | | 0 | min | | Max cumulative flush time across primary shards | | 0 | min | | Total Young Gen GC time | | 0.062 | s | | Total Young Gen GC count | | 5 | | | Total Old Gen GC time | | 0 | s | | Total Old Gen GC count | | 0 | | | Store size | | 3.27006e-05 | GB | | Translog size | | 0.0315683 | GB | | Heap used for segments | | 0.365074 | MB | | Heap used for doc values | | 0.111397 | MB | | Heap used for terms | | 0.219269 | MB | | Heap used for norms | | 0.0147095 | MB | | Heap used for points | | 0 | MB | | Heap used for stored fields | | 0.0196991 | MB | | Segment count | | 40 | | | Min Throughput | bulk | 1100.03 | docs/s | | Median Throughput | bulk | 2616.48 | docs/s | | Max Throughput | bulk | 6598.72 | docs/s | | 50th percentile latency | bulk | 727.761 | ms | | 90th percentile latency | bulk | 2123.67 | ms | | 100th percentile latency | bulk | 2156.42 | ms | | 50th percentile service time | bulk | 727.761 | ms | | 90th percentile service time | bulk | 2123.67 | ms | | 100th percentile service time | bulk | 2156.42 | ms | | error rate | bulk | 0 | % | -------------------------------- [INFO] SUCCESS (took 56 seconds) -------------------------------- ``` Et voilà! Data has been imported in your brand new cluster. 🙂 ---
**Author:** ![Joseph\_Presseller](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_presseller/32/69330_2.png) [@Joseph\_Presseller](https://discuss.elastic.co/u/Joseph_Presseller)\ **Post date:** [December 19, 2020, 8:50pm UTC](https://discuss.elastic.co/t/dec-7th-2020-en-using-rally-as-a-data-import-export-tool/257820/2 "2020-12-19T20:50:51Z")
Laurent - Thanks for this post, I used it several time these past couple of weeks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 16, 2021, 8:51pm UTC](https://discuss.elastic.co/t/dec-7th-2020-en-using-rally-as-a-data-import-export-tool/257820/3 "2021-01-16T20:51:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.