Create track using datastreams

[kvmuralidhar (Murali V)]

Hi,

I am trying to create a custom track using an data stream index and it's failing because of the datastream is an hidden index. What's the work around to create this custom track ?

Thanks
Murali

[Bradley_Deam]

Hey @kvmuralidhar !

Unfortunately we don't have this functionality available - we are aware of it though: Support data streams in create-track · Issue #1360 · elastic/rally · GitHub

However, you can 'hack' Rally for a temp workaround (I quickly tested this locally) by checking out the repo locally and modifying..

Line 68:

# Original
response = client.indices.get(index_pattern)

# Modified to
response = client.indices.get(index_pattern, params={"expand_wildcards": "all"})

Line 52-53:

# Original
    if index_name.startswith("."):
        return False, f"Index [{index_name}] is hidden"
# Modified to (comment these lines out)
    # if index_name.startswith("."):
    #     return False, f"Index [{index_name}] is hidden"

That should unblock you for the time being. I tested this with my own personal cluster:

$ esrally create-track --track=iis --target-hosts=https://my-es-cluster:9243 --client-options="timeout:320,use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'not a real password'" --indices=".ds-metrics-windows.service-default-2021.05.21-000013" --output-path=.


    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Connected to Elasticsearch cluster [instance-0000000068] version [7.12.0].

Extracting documents for index [.ds-metrics-windows.se...     1000/1000 docs [100.0% done]
Extracting documents for index [.ds-metrics-window...     216117/1947230 docs [11.1% done]

[kvmuralidhar (Murali V)]

@Bradley_Deam

It worked like a champ, thanks for the work around.

Murali