How to source search queries from a file

[username123]

Let's say I have a huge file with the queries that I have recorded over time:

"GET /primary/_analyze?analyzer=foldedlower&text=yomu
"GET /primary/_analyze?analyzer=foldedlower&text=yonde
"GET /primary/_analyze?analyzer=foldedlower&text=Yo+oigo+con+mis+orejas
"GET /primary/_analyze?analyzer=foldedlower&text=Yo+tengo+dos+ojos
"GET /primary/_analyze?analyzer=foldedlower&text=You%27re+eligible

I want to be able to point my query operation type to this file, so that the queries created by it would be sourced from the file. I can clean the data out and have everything after &text= utilized, but I am not certain how I would plug it into configuration. What would be a correct path to take?

[danielmitterdorfer]

Hi,

I think this is best solved by implementing a so-called parameter source. There are two options to implement them: either as a (Python) function or as a class. In your case I'd go for a class. In the constructor you can read the file and prepare a suitable data structure (e.g. store the queries in a list) and in the params() method you just pick a query randomly or iterate through them - whatever suits your use case. In our official tracks, we to something very similar in the geonames track which you can use as a starting point for your own parameter source.

Note that the params() method of the parameter source is called on a performance-critical path so you should do any preprocessing of the file already in the constructor, otherwise you might introduce an accidental bottleneck in the load generator. You can use Rally's profiling support to profile it and double-check that there are no problems.

Daniel

[username123]

I see!

Thank you Daniel! That was pretty much the direction I was thinking of going.