Query parameters for operation-type search

Elastic Stack Elasticsearch
1

Is it possible to specify one or more query parameters for search operations within track.json?

In my case I'm benchmarking a plug-in that is activated via a query parameter that I defined, e.g.:

GET /_search?my_custom_param=foo
{
"query": {
"match_all": {}
}
}

Is this possible?

Thanks!

2

Hi @bherring,

since Rally 0.6.1 this is indeed possible. It has been implemented in #290 where you will find a bit more details.

In your case the syntax is:

    {
      "name": "default",
      "operation-type": "search",
      "body": {
        "query": {
          "match_all": {}
        }
      },
      "request-params": {
        "my_custom_param": "foo",
      }
    }

Daniel

3

Thanks for the quick response!

I upgraded to 0.6.2 and added the "request-params" specifier to my track.json as you described, and re-ran my race.

An exception was raised at racecontrol.py, line 164 related to "unexpected keyword", as if it doesn't like my custom query parameter. I double-checked that the same query is working fine when requested outside of esrally.

Below is a snippet from the generated log showing the exception. I'd be happy to send you the full log but it is too long to paste inline and the upload feature doesn't allow text files.

Thanks again!

2017-07-18 14:38:04,928 PID:61499 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 241, in race
may_continue = benchmark.run(lap)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 164, in run
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ('Error in load generator [0]', TypeError("search() got an unexpected keyword argument 'rba_list'",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/m2/.local/lib/python3.4/site-packages/esrally/rally.py", line 446, in dispatch_sub_command
race(cfg)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/rally.py", line 380, in race
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/rally.py", line 400, in with_actor_system
runnable(cfg)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/rally.py", line 380, in
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 340, in run
raise e
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 337, in run
pipeline(cfg)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 42, in call
self.target(cfg)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 285, in benchmark_only
return race(Benchmark(cfg, external=True))
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 250, in race
benchmark.teardown(cancelled, error)
File "/home/m2/.local/lib/python3.4/site-packages/esrally/racecontrol.py", line 183, in teardown
raise exceptions.RallyError("Mechanic has not stopped engine but instead [%s]. Terminating race without result." % str(result))
esrally.exceptions.RallyError: Mechanic has not stopped engine but instead [<esrally.driver.driver.BenchmarkFailure object at 0x7f4efe0024a8>]. Terminating race without result.

4

Hi @bherring,

I could reproduce the issue.

The reason Rally is using the official Elasticsearch Python client to issue requests against Elasticsearch. Prior to creating the request, the client checks that the passed request parameters match one of the parameters in the documentation. As your request parameter rba_list is a non-standard one, it fails the validation and thus the benchmark fails.

I see if I can get around that limitation somehow but I fear it's gonna take a little while.

If it's very important to you, you can write a custom runner for this operation and basically use a low-level method of the Python client.

Something along these lines should do what you want:

# instead of self.es.search(...) do:
import elasticsearch.client
self.es.transport.perform_request("GET", elasticsearch.client._make_path(index, type, '_search'), params=your_request_parameter_map_here, body=body)

but obviously this is more fragile than relying on the standard mechanisms.

Daniel

5

Thank you Daniel that is very helpful. If there is a way around this from your side I think that would be worth doing (for future search plug-in developers like me). Meanwhile I'll try out your suggestion of creating a custom runner.

6

Daniel your suggestion worked great, thanks!

Here is what my track.py looks like, just in case this might be useful to others....

import elasticsearch.client

def search_custom(es, params):
es.transport.perform_request("GET",
elasticsearch.client._make_path(
params["index"],
params["type"],
'_search'),
params=params["request-params"],
body=params["body"]
)

def register(registry):
registry.register_runner("search_custom", search_custom)

7

Hi Brian,

for the moment I'd stick to the custom runner as this gives you the most control (glad to hear that it works :slight_smile: ). To allow this in Rally core at some point I've raised #302 in the Rally repo.

Daniel

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.