Bulk index operation for multiple indices

[Phuong_Hoang]

I am using esrally version 0.9.1, elasticsearch version 6.1.3
After executing the custom track for multiple indices, I can index documents for only oci-vcn but not oci-user(testing with the existing es-cluster)

How could I index the documents for all indices (both oci-vcn, oci-user) ?

This is the track.json

{
  "name": "Sample track",
  "indices": [
    {
      "name": "oci-user",
      "auto-managed": false,
      "body": "user-index.json",
      "types": [ "oci-user" ]
    },
    {
      "name": "oci-vcn",
      "auto-managed": false,
      "body": "vcn-index.json",
      "types": [ "oci-vcn" ]
    }
  ],
  "corpora": [
    {
      "name": "oci-vcn",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 1,
          "target-type": "oci-vcn",
          "target-index": "oci-vcn"
        }
      ]
    },
    {
      "name": "oci-user",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 1,
          "target-type": "oci-user",
          "target-index": "oci-user"
        }
      ]
    }
  ],
  "challenges": [
    {
      "name": "create-index",
      "default": true,
      "schedule": [
        {
          "operation": {
            "operation-type": "delete-index"
          }
        },
        {
          "operation": {
            "operation-type": "create-index"
          }
        },
        {
          "operation": {
            "operation-type": "cluster-health",
            "request-params": {
              "wait-for-status": "green"
            }
          }
        },
        {
          "operation": {
            "operation-type": "bulk",
            "bulk-size": 10
          }
        }
      ]
    }
  ]
}

This is user-index.json

{
  "settings": {
    "number_of_replicas": 0
  },
  "mappings": {
    "oci-user": {
      "dynamic": "strict",
      "properties": {
        "name": {
          "type": "text"
        },
        "age": {
          "type": "integer"
        },
        "email": {
          "type": "keyword"
        }
      }
    }
  }
}

This is vcn-index.json

{
  "settings": {
    "number_of_replicas": 0
  },
  "mappings": {
    "oci-vcn": {
      "dynamic": "strict",
      "properties": {
        "name": {
          "type": "text"
        }
      }
    }
  }
}

This is oci-user/documents.json

{ "name": "Foo Bar", "age": 40, "email": "Foo@Bar.com" }

This is oci-vcn/documents.json

{ "name": "Foo Bar"}

[danielmitterdorfer]

Hi @Phuong_Hoang,

you have fallen into a very subtle trap (see already open issue #377). If you do not specify any warmup time-period, Rally assumes that by default you want to run an iteration-based benchmark. The default is to run one iteration and therefore the benchmark only issues one bulk request.

You can fix this by specifying a warmup time-period (even if it just one second). E.g:

{
    "operation": {
        "operation-type": "bulk",
        "bulk-size": 10
    },
    "warmup-time-period": 120
}

I have to say that this is extremely hard to spot and we should do something about it. Hence, I have increased the priority of the ticket that I've mentioned.

Daniel