Rally create-track limit document count

[mots]

I am able to create a custom track following these instructions.

But the target index I am using has over 5 mil documents. It's taking too long to create the custom track and I don't even need that many documents to run my test. I would like to restrict the number of documents added to the custom track to say 5k.

How can I achieve this ?

[Evgenia_Badiyanova]

Hello,

If you are using bulk operation to index those documents, you can use optional bulk operation property ingest-percentage in your track.

Thanks,
Evgenia

[mots]

Thanks for replying back.

But my issue is related to creating the custom track.

esrally create-track --track=acme --target-hosts=abcdef123.us-central-1.gcp.cloud.es.io:9243 --client-options="timeout:60,use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'secret-password'" --indices="products,companies" --output-path=~/tracks

I am using the above command, just changing the args to point to my target index. This target index has over 5 mil documents. The above command tries to create the custom track with all the 5 mill documents. I would like to limit it to only 5000 documents or % of my choosing.

Is this possible ?

[mots]

I am getting around this use case by doing the following -

POST _reindex?wait_for_completion=false
{
  "max_docs": 5000,
  "source": {
    "index": "esrally-documents",
    "query": {
      "function_score" : {
        "random_score" : {},
        "min_score" : 0.9    
      }
    }
  },
  "dest": {
    "index": "esrally-5k-documents"
  }
}

esrally-documents index has 5 million documents. I am using the reindex api to create a new index esrally-5k-documents with 5000 documents. Then running the create-track on this new index.

@danielmitterdorfer @Evgenia_Badiyanova Any feedback ?

[RickBoyd]

This is currently your best option, in my opinion. Note that 5k documents may not be enough to reliably reproduce benchmark results, but as you already know you can get more later with this same method.