# Multiple indices are indexing in sequence

**URL:** <https://discuss.elastic.co/t/multiple-indices-are-indexing-in-sequence/175462>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [April 4, 2019, 6:18pm UTC](https://discuss.elastic.co/t/multiple-indices-are-indexing-in-sequence/175462 "2019-04-04T18:18:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![suvarna](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@suvarna](https://discuss.elastic.co/u/suvarna)\
**Post date:** [April 4, 2019, 6:18pm UTC](https://discuss.elastic.co/t/multiple-indices-are-indexing-in-sequence/175462/1 "2019-04-04T18:18:29Z")

</div>

Hi

I am using Rally's existing tracks to perform benchmarking. I noticed that nyc taxis is the largest track with 4.5GB compressed and 74.3 GB uncompressed docs. I want to test with larger data volume.

I have used the following trick..  
the nyc\_taxis document corpus ten times (note the `index_count` variable at the top):

```auto
{% set index_count = 10 %}
{
  "version": 2,
  "description": "Taxi rides in New York in 2015",
  "indices": [
  {% set comma = joiner() %}
  {% for item in range(index_count) %}
  {{ comma() }}
    {
      "name": "nyc_taxis-{{item}}",
      "body": "index.json",
      "types": ["type"],
      "auto-managed": false
    }
  {% endfor %}
  ],
  "corpora": [
    {
      "name": "nyc_taxis",
      "base-url": "http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/nyc_taxis",
      "documents": [
      {% set comma = joiner() %}
      {% for item in range(index_count) %}
      {{ comma() }}

```

i have referred below link for above trick.

> [@Increase data size in Rally existing tracks](https://discuss.elastic.co/t/increase-data-size-in-rally-existing-tracks/116514/2):
>
> Hi @Alp1, you can apply a trick so Rally indexes the data into multiple indices but you need to create your own track for that. I suggest that you use the latest version (which is 0.9.1) because we introduced a concept of "document corpora" recently with Rally 0.9.0. This feature allows you to reuse document corpora from other tracks. Here is a complete example that bulk-indexes the nyc\_taxis document corpus ten times (note the index\_count variable at the top): {% set index\_count = 10 %} { "…

My Concern is: when i used above trick .. the ES will have 10 different indices and those are running in sequence ..

how to make that indices to run in parallel so that it can utilize the CPU in an optimal way.

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [April 9, 2019, 12:14pm UTC](https://discuss.elastic.co/t/multiple-indices-are-indexing-in-sequence/175462/2 "2019-04-09T12:14:19Z")

</div>

> [@suvarna](#):
>
> My Concern is: when i used above trick .. the ES will have 10 different indices and those are running in sequence ..
> 
> how to make that indices to run in parallel so that it can utilize the CPU in an optimal way.

All specified clients will send bulk requests to Elasticsearch as fast as they can, i.e. you control that by varying the number of clients instead of the number of indices that you bulk-index into unless I misunderstand what you're after. Hope that helps. 🙂

Daniel

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2019, 12:14pm UTC](https://discuss.elastic.co/t/multiple-indices-are-indexing-in-sequence/175462/3 "2019-05-07T12:14:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.