The results of the stress tests for Bool query combine filter were abnormal

[Lijian_Liao]

I used rally to do stress tests , and i found one of the result was abnormal.I was confused .The query combined filter 's latency and service time was higher so much than must .

I think filter should be more efficient than must。

the track.py:

the challenge：

I also used JMeter to verify this, and the results were the opposite of Rally's。

[RickBoyd]

I don't have any initial answers for you, but a few questions/suggestions based on the available information.

1. What do the operation definitions look like?
2. Does the outcome change if you change the order of execution of the tasks in your challenge schedule?
3. To remove the "noise" of having very high latency measurements can you remove or drastically lower your target-throughput? See the docs for an explanation of why those numbers (latency, service time) are so different from one another in the current mode for the same operation
4. Can you elaborate on in what way specifically JMeter's results were opposite to Rally?

Thanks,
Rick B

[Lijian_Liao]

This is my operation for two query in picture.
{
"name": "randomized-must-term-queries",
"operation-type": "search",
"param-source": "must-term-query-source"
},
{
"name": "randomized-filter-queries",
"operation-type": "search",
"param-source": "filter-query-source"
},

And I registry them in track.py:
registry.register_param_source("must-term-query-source", MustTermQueryParamSource)
registry.register_param_source("filter-query-source", FilterQueryParamSource)

Finally I lower my target-throughput and change the order in a new challenge.
{
"name": "nested-search-challenge-must-filter",
"description": "Indexes the document corpus for an hour using Elasticsearch default settings. After that randomized nested queries are run.",
"schedule": [
{
"operation": "randomized-must-term-queries",
"clients": 20,
"warmup-iterations": 100,
"target-throughput": 1200,
"iterations": 100
},
{
"operation": "randomized-filter-queries",
"clients": 20,
"warmup-iterations": 100,
"target-throughput": 1200,
"iterations": 100
}
]
},
{
"name": "nested-search-challenge-filter-must",
"description": "Indexes the document corpus for an hour using Elasticsearch default settings. After that randomized nested queries are run.",
"schedule": [
{
"operation": "randomized-filter-queries",
"clients": 20,
"warmup-iterations": 100,
"target-throughput": 1200,
"iterations": 100
},
{
"operation": "randomized-must-term-queries",
"clients": 20,
"warmup-iterations": 100,
"target-throughput": 1200,
"iterations": 100
}
]
}

the results:
| Min Throughput | randomized-filter-queries | 466.17 | ops/s |
| Median Throughput | randomized-filter-queries | 471.78 | ops/s |
| Max Throughput | randomized-filter-queries | 498.06 | ops/s |
| 50th percentile latency | randomized-filter-queries | 3947.52 | ms |
| 90th percentile latency | randomized-filter-queries | 4905.27 | ms |
| 99th percentile latency | randomized-filter-queries | 5104.14 | ms |
| 99.9th percentile latency | randomized-filter-queries | 5194.24 | ms |
| 100th percentile latency | randomized-filter-queries | 5227.62 | ms |
| 50th percentile service time | randomized-filter-queries | 39.8017 | ms |
| 90th percentile service time | randomized-filter-queries | 95.458 | ms |
| 99th percentile service time | randomized-filter-queries | 145.216 | ms |
| 99.9th percentile service time | randomized-filter-queries | 164.565 | ms |
| 100th percentile service time | randomized-filter-queries | 183.332 | ms |
| error rate | randomized-filter-queries | 0 | % |
| Min Throughput | randomized-must-term-queries | 290.23 | ops/s |
| Median Throughput | randomized-must-term-queries | 296.37 | ops/s |
| Max Throughput | randomized-must-term-queries | 298.93 | ops/s |
| 50th percentile latency | randomized-must-term-queries | 7780.84 | ms |
| 90th percentile latency | randomized-must-term-queries | 9675.04 | ms |
| 99th percentile latency | randomized-must-term-queries | 10028.1 | ms |
| 99.9th percentile latency | randomized-must-term-queries | 10198.7 | ms |
| 100th percentile latency | randomized-must-term-queries | 10242.7 | ms |
| 50th percentile service time | randomized-must-term-queries | 62.0603 | ms |
| 90th percentile service time | randomized-must-term-queries | 129.571 | ms |
| 99th percentile service time | randomized-must-term-queries | 193.743 | ms |
| 99.9th percentile service time | randomized-must-term-queries | 219.965 | ms |
| 100th percentile service time | randomized-must-term-queries | 226.663 | ms |
| error rate | randomized-must-term-queries | 0 | % |

| Min Throughput | randomized-must-term-queries | 287.78 | ops/s |
| Median Throughput | randomized-must-term-queries | 294.97 | ops/s |
| Max Throughput | randomized-must-term-queries | 297.39 | ops/s |
| 50th percentile latency | randomized-must-term-queries | 7833.24 | ms |
| 90th percentile latency | randomized-must-term-queries | 9731.55 | ms |
| 99th percentile latency | randomized-must-term-queries | 10131.2 | ms |
| 99.9th percentile latency | randomized-must-term-queries | 10297.9 | ms |
| 100th percentile latency | randomized-must-term-queries | 10343.2 | ms |
| 50th percentile service time | randomized-must-term-queries | 61.9679 | ms |
| 90th percentile service time | randomized-must-term-queries | 131.14 | ms |
| 99th percentile service time | randomized-must-term-queries | 204.717 | ms |
| 99.9th percentile service time | randomized-must-term-queries | 238.061 | ms |
| 100th percentile service time | randomized-must-term-queries | 259.852 | ms |
| error rate | randomized-must-term-queries | 0 | % |
| Min Throughput | randomized-filter-queries | 466.22 | ops/s |
| Median Throughput | randomized-filter-queries | 472.71 | ops/s |
| Max Throughput | randomized-filter-queries | 500.44 | ops/s |
| 50th percentile latency | randomized-filter-queries | 3947.12 | ms |
| 90th percentile latency | randomized-filter-queries | 4911.61 | ms |
| 99th percentile latency | randomized-filter-queries | 5045.84 | ms |
| 99.9th percentile latency | randomized-filter-queries | 5120.61 | ms |
| 100th percentile latency | randomized-filter-queries | 5153.45 | ms |
| 50th percentile service time | randomized-filter-queries | 39.9451 | ms |
| 90th percentile service time | randomized-filter-queries | 96.0194 | ms |
| 99th percentile service time | randomized-filter-queries | 144.568 | ms |
| 99.9th percentile service time | randomized-filter-queries | 166.275 | ms |
| 100th percentile service time | randomized-filter-queries | 170.964 | ms |
| error rate | randomized-filter-queries | 0 | % |

I tested the previous one：
| Min Throughput | randomized-filter-queries | 1124.47 | ops/s |
| Median Throughput | randomized-filter-queries | 1145.52 | ops/s |
| Max Throughput | randomized-filter-queries | 1172.26 | ops/s |
| 50th percentile latency | randomized-filter-queries | 12474.8 | ms |
| 90th percentile latency | randomized-filter-queries | 17317 | ms |
| 99th percentile latency | randomized-filter-queries | 18447.2 | ms |
| 99.9th percentile latency | randomized-filter-queries | 18766 | ms |
| 99.99th percentile latency | randomized-filter-queries | 18879.6 | ms |
| 100th percentile latency | randomized-filter-queries | 18902.9 | ms |
| 50th percentile service time | randomized-filter-queries | 155.22 | ms |
| 90th percentile service time | randomized-filter-queries | 280.322 | ms |
| 99th percentile service time | randomized-filter-queries | 406.64 | ms |
| 99.9th percentile service time | randomized-filter-queries | 508.57 | ms |
| 99.99th percentile service time | randomized-filter-queries | 635.337 | ms |
| 100th percentile service time | randomized-filter-queries | 743.608 | ms |
| error rate | randomized-filter-queries | 0 | % |
| Min Throughput | randomized-must-term-queries | 644.66 | ops/s |
| Median Throughput | randomized-must-term-queries | 657.09 | ops/s |
| Max Throughput | randomized-must-term-queries | 675.92 | ops/s |
| 50th percentile latency | randomized-must-term-queries | 25574.3 | ms |
| 90th percentile latency | randomized-must-term-queries | 34686.7 | ms |
| 99th percentile latency | randomized-must-term-queries | 36764.9 | ms |
| 99.9th percentile latency | randomized-must-term-queries | 37062.7 | ms |
| 99.99th percentile latency | randomized-must-term-queries | 37222.6 | ms |
| 100th percentile latency | randomized-must-term-queries | 37242.5 | ms |
| 50th percentile service time | randomized-must-term-queries | 268.461 | ms |
| 90th percentile service time | randomized-must-term-queries | 452.616 | ms |
| 99th percentile service time | randomized-must-term-queries | 655.654 | ms |
| 99.9th percentile service time | randomized-must-term-queries | 809.379 | ms |
| 99.99th percentile service time | randomized-must-term-queries | 905.664 | ms |
| 100th percentile service time | randomized-must-term-queries | 1021.44 | ms |
| error rate | randomized-must-term-queries | 0 | % |

OK ,the same with jmeter ，filter performs better.I will figure out what happen previous test...

Thanks a lot