# Benchmarking ES cluster using larger Rally dataset for multiple parallel indexing **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [June 6, 2019, 1:09am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477 "2019-06-06T01:09:36Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![cks](https://avatars.discourse-cdn.com/v4/letter/c/8edcca/32.png) [@cks](https://discuss.elastic.co/u/cks)\ **Post date:** [June 6, 2019, 1:09am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477/1 "2019-06-06T01:09:37Z")
> [@Increase data size in Rally existing tracks](https://discuss.elastic.co/t/increase-data-size-in-rally-existing-tracks/116514/2): > > Hi @Alp1, you can apply a trick so Rally indexes the data into multiple indices but you need to create your own track for that. I suggest that you use the latest version (which is 0.9.1) because we introduced a concept of "document corpora" recently with Rally 0.9.0. This feature allows you to reuse document corpora from other tracks. Here is a complete example that bulk-indexes the nyc\_taxis document corpus ten times (note the index\_count variable at the top): {% set index\_count = 10 %} { "… The above post allowed me to create 10x indices for nyc\_taxis dataset, however only the shards for one index were populated in parallel, then rolls over to next index and so on. Is there a way to populate all 10 indices in parallel? **env:** - 4x ES nodes, 512gb ram, 48 vcores, 30TB Flash NAS storage. - 1-4x Rally nodes,( 3x load generators) - Track: nyc\_taxis. - Heap is 31GB **bulk\_indexing\_client=40, num\_of\_shards=40, bulk\_size=175000** I got about 792k docs/s for index-append running on 1x rally node and 4x ES nodes. CPU saturation was 50-60% on each ES nodes. Increasing number of load generators did not seem increase docs/s number. I am trying to max out cpu and storage IO. Any other ways to stress test ES nodes? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 6, 2019, 6:18am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477/2 "2019-06-06T06:18:03Z")
What is the use-case you are benchmarking for? What type of data will you be indexing? Will you be using time-based indices? If so - do you have a specified retention period? What is the aim of your benchmark? Given the size of the host I would probably recommend running multiple nodes per host. ---
**Author:** ![cks](https://avatars.discourse-cdn.com/v4/letter/c/8edcca/32.png) [@cks](https://discuss.elastic.co/u/cks)\ **Post date:** [June 7, 2019, 7:06am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477/3 "2019-06-07T07:06:54Z")
At this time looking to benchmark Elastic for high throughput indexing. The likely use case is apache logs. The record size in http\_logs track seem to be light, hence was trying on nyc\_taxis track. Not particularly looking at time based indices, may be in future. Retention is likely 6 months. Aim is to maximize ElasticSearch performance on available physical environment. I had setup another 4 ES nodes with lesser Cpu and ram. 32 vcores and 256 GB RAM - index-append came down by 20% when compared with 48vcores and 512GB ram. heap was still 31 gb for both, all other params were same. Here is what I am trying: 1. find optimal configuration for full indexing (bulk) for cluster. 2. find optimal configuration for searches. 3. find optimal configuration for cluster with both search and indexing at same time. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 7, 2019, 7:18am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477/4 "2019-06-07T07:18:02Z")
For that type of data you almost always want to use time-based indices, e.g. through rollover together with ILM. This generally means that you are indexing into a few shards per node at most at any time. I would still recommend setting up multiple nodes per host. You could also look at the rally-eventdata-track which was designed to simulate this use case and then adapt challenges to fit you needs. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 7, 2019, 7:35am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477/5 "2019-06-07T07:35:02Z")
Also have a look at the following resources: [https://www.elastic.co/es/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right](https://www.elastic.co/es/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right) [https://www.elastic.co/es/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/es/elasticon/conf/2016/sf/quantitative-cluster-sizing) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2019, 7:35am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-using-larger-rally-dataset-for-multiple-parallel-indexing/184477/6 "2019-07-05T07:35:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.