# Esrally ingesting to two indices parallelly

**URL:** <https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [July 29, 2021, 5:36am UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933 "2021-07-29T05:36:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sameera\_De\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sameera_de_silva/32/91815_2.png) [@Sameera\_De\_Silva](https://discuss.elastic.co/u/Sameera_De_Silva)\
**Post date:** [July 29, 2021, 5:36am UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/1 "2021-07-29T05:36:03Z")

</div>

I want to add documents to two indices simultaneously. For that, I tried with parallel option . But it stills run sequentially and add data to firstly mentioned index and later to the second one . Is this supported is Esrally ? If so could you please guide me . I already managed for the parallel search.

Below is my track.json

```auto
  "version": 2,
  "description": "Tutorial benchmark for Rally",
  "indices": [
    {
      "name": "customrecords",
      "body": "",
      "types": [
        "docs"
      ]
    }
  ],
  "corpora": [
    {
      "name": "rally-tutorial",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 12109130,
          "uncompressed-bytes": 12258880607,
          "target-index": "geocustom"
        },
        {
          "source-file": "samples.json",
          "document-count": 12218269,
          "uncompressed-bytes": 7440925751,
          "target-index": "customrecords"
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        }
      }
    },
    {
      "parallel": {
        "tasks": [
          {
            "operation": {
              "operation-type": "bulk",
              "bulk-size": 100
            },
            "warmup-time-period": 120,
            "clients": 8,
            "target-throughput": 800
          }
        ]
      }
    }
  ]
}
type or paste code here

```

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [July 29, 2021, 7:39am UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/2 "2021-07-29T07:39:10Z")

</div>

Please check the docs: [bulk operation](https://esrally.readthedocs.io/en/stable/track.html#properties), [corpora](https://esrally.readthedocs.io/en/stable/track.html#corpora) and an example in the http\_logs standard track: [indices+corpora](https://github.com/elastic/rally-tracks/blob/e7fa3f9ae1f77535e30896d257bae3f4a6d78227/http_logs/track.json#L13-L103) and [index-append operation definition](https://github.com/elastic/rally-tracks/blob/e7fa3f9ae1f77535e30896d257bae3f4a6d78227/http_logs/operations/default.json#L1-L7).

---

<div class="post-metadata">

**Author:** ![Sameera\_De\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sameera_de_silva/32/91815_2.png) [@Sameera\_De\_Silva](https://discuss.elastic.co/u/Sameera_De_Silva)\
**Post date:** [July 29, 2021, 10:35am UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/3 "2021-07-29T10:35:23Z")

</div>

> [@dliappis](#):
>
> corpus

Thank you , I after revisiting the suggestion, I modified the track,json as below. Sadly, still its ingest data sequentially. Kindly help.

```auto
{
  "version": 2,
  "description": "Tutorial benchmark for Rally",
  "indices": [
    {
      "name": "customrecords",
      "body": "",
      "types": ["docs"]
    },
	    {
      "name": "geocustom",
      "body": "",
      "types": ["docs"]
    }
  ],
  "corpora": [
    {
      "name": "rally-tutorial",
      "documents": [
        {
          "source-file": "samples.json",
          "document-count": 100000,
          "uncompressed-bytes": 3295600000,
          "target-index": "geocustom"
        }
		,
      {
        "source-file": "samples.json",
        "document-count": 100000,
	    "uncompressed-bytes": 3295600000,
        "target-index": "customrecords"
      }
      ]
    }
  ],
  "schedule": [
 {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        }
      }
    },
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 100
      },     
      "warmup-time-period": 120,
      "clients": 8
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [July 29, 2021, 11:53am UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/4 "2021-07-29T11:53:35Z")

</div>

> [@Sameera\_De\_Silva](#):
>
> Thank you , I after revisiting the suggestion, I modified the track,json as below. Sadly, still its ingest data sequentially. Kindly help.

Sorry, I misunderstood your original question.

The strategy I mentioned earlier will work for corpora that include the [ACTION\_AND\_METADATA](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk) line which specifies the target index. So if you could have one corpus file that contains ALL docs that should be ingested, and specify `"includes-action-and-meta-data": true` in the corpora section you can achieve what you want with just one corpus file.

However, if you must work with separate corpus docs, you can use the parallel approach you [mentioned earlier](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933). This example was missing two parallel `bulk` tasks. I came up with the following quick example which does what you described: (btw I created the two corpus files from the respective corpora of [geonames](https://github.com/elastic/rally-tracks/blob/e7fa3f9ae1f77535e30896d257bae3f4a6d78227/geonames/track.json#L12-L25) and [http\_logs](https://github.com/elastic/rally-tracks/blob/e7fa3f9ae1f77535e30896d257bae3f4a6d78227/http_logs/track.json#L47-L59) tracks using `head -100000 ~/.rally/benchmarks/data/geonames/documents-2.json >geodocs.json` and `head -100000 ~/.rally/benchmarks/data/http_logs/documents-181998.json >logsdocs.json`):

```auto
{
  "version": 2,
  "description": "Tutorial benchmark for Rally",
  "indices": [
    {
      "name": "customlogs",
      "body": ""
    },
    {
      "name": "customgeo",
      "body": ""
    }
  ],
  "corpora": [
    {
      "name": "rally-tutorial",
      "documents": [
        {
          "source-file": "./geodocs.json",
          "document-count": 100000,
          "target-index": "customgeo"
        },
        {
          "source-file": "./logsdocs.json",
          "document-count": 100000,
          "target-index": "customlogs"
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "yellow"
        }
      }
    },
    {
      "parallel": {
        "tasks": [
          {
            "name": "bulk1",
            "operation": {
              "operation-type": "bulk",
              "indices": "customgeo",
              "bulk-size": 100
            },
            "warmup-time-period": 0,
            "clients": 1,
            "target-throughput": 800
          },
          {
            "name": "bulk2",
            "operation": {
              "operation-type": "bulk",
              "indices": "customlogs",
              "bulk-size": 100
            },
            "warmup-time-period": 0,
            "clients": 1,
            "target-throughput": 800
          }
        ]
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Sameera\_De\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sameera_de_silva/32/91815_2.png) [@Sameera\_De\_Silva](https://discuss.elastic.co/u/Sameera_De_Silva)\
**Post date:** [July 29, 2021, 1:25pm UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/5 "2021-07-29T13:25:27Z")

</div>

Thank you very much. it worked like a charm, you saved me from a lot of trouble.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/d/0d2e66f88c262bd18e9df7d8e0859f797d01622c.png)

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [July 29, 2021, 1:44pm UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/6 "2021-07-29T13:44:57Z")

</div>

> [@Sameera\_De\_Silva](#):
>
> Thank you very much. it worked like a charm, you saved me from a lot of trouble.

You are welcome @Sameera_De_Silva, that's great to hear.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 26, 2021, 1:45pm UTC](https://discuss.elastic.co/t/esrally-ingesting-to-two-indices-parallelly/279933/7 "2021-08-26T13:45:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.