ESRally Not Responding Search Queries if it exceeds from 280 Clients

Elastic Stack Elasticsearch
1

Hi,
I am doing the Elasticsearch -6.6.2 Search Queries Benchmark on NVMe device(Non-Volatile Memory).
I am having mixed queries of Simple, Complex and moderate.
But ESRally not responding if it exceeds 280 clients. Below is my challenges configured in elasticlogs-querying.json.

  {

"name": "elasticlogs-querying",
"description": "This challenge simulates a set of Kibana queries against historical data (elasticlogs_q-* indices) without any indexing taking place for a period of 30 minutes. It assumes one of the challenges creating elasticlogs_q-* indices has been run.",
"meta": {
"benchmark_type": "querying",
"target_kibana_queries_per_minute": 290
},
"schedule": [
{
"operation": "fieldstats_elasticlogs_q-*",
"iterations": 1,
"clients": 290
},
{
"parallel": {
"warmup-time-period": 0,
"time-period": 600,
"clients": 290,
"tasks": [
{
"operation": "relative-kibana-content_issues-dashboard_50%",
"target-interval": 30
},

So, just want to know, is there any restrictions on ESRally side?
Also, its one node cluster.
Any help will be really appreciated.

--Regards,
Balmukund

2

I think you are completely overloading the cluster. What are your requirements?

3

Hi Christian,
Thank you for your quick response. My requirement is to test with Millions of queries of mixed type i.e. simple, complex and moderate type. Below is my query types as well other types such as Range based, Fuzzy, Wildcard, Regexp etc.

Simple Query:

{
"name": "simple_query",
"operation-type": "search",
"index": "elasticlogs_*",
"body": {
"query": {
"nginx.access.geoip.country_iso_code: AT"
}
}
}

Complex Query:

{
"name": "complex_query",
"operation-type": "search",
"index": "elasticlogs_*",
"body": {
"query": {"bool":
{"must": [{"bool": {"should": [
{"match": {"nginx.access.geoip.country_name": ""}},
{"match": {"nginx.access.geoip.country_iso_code": ""}}
]}},
{"bool": {"should": [
{"match": {"nginx.access.user_agent.os_name": "Mac OS X"}},
{"match": {"nginx.access.user_agent.device": "Other"}}
]}},
{"bool": {"should": [
{"match": {"nginx.access.geoip.continent_name": "Asia"}},
{"match": {"nginx.access.response_code": "200"}}
]}},
{"bool": {"should": [{"match": {"nginx.access.geoip.city_name": ""}},
{"match": {"nginx.access.remote_ip": "203.112.82.0"}}
]}}
]}}
}
}

By using above combinations, I want to use 70% Simple query and 30% other queries and maximize CPU Utilization at least 80%.
But I could see, ESRally is not responding after 280 such combined requests.
Please let me know if you required any other details.

--Regards,
Balmukund

4

Start with a low level of concurrent requests and increase slowly until you find the limit of the cluster. Hitting and expecting a single node cluster to cope with that many events of varying complexity seems pointless.

5

Hi Christian,
Thank you very much for your reply.Firstly, i tried with only complex queries as given above which uses ~82% CPU but number of such requests are allowed only 50. So, how to achieve such requests in Millions as below referral link.

Here, each type is in Million.

--Regards,
Balmukund

6

What is your requirement? How much data will you have and how many queries per second do you need to support? What is the expected query mix and latency requirements? Will you at the same time be indexing and/or updating data? If so, what is the expected index/update rate?

Once you have a grip of the answer to these questions you can start benchmarking in order to find the ideal cluster size and configuration. A singlenode cluster has limited amount of resources so may not be sufficient for the query volumes you are mentioning.

7

Hi Christian,
Thank you for your reply. My main aim is to maximize %CPU Utilization. So, We are trying to use maximum simple queries.So, whenever i tried using 300 Simple Queries, ESRally stop working and it goes into hung state.
It does not do indexing and/or updating data parallel.
As, per your previous suggestion to use only one category query. So, i am using above mentioned simple queries only. If the number of simple queries less than 300, it gives the CPU Utilization below 1%. Below is my system configurations.
Heap Size: 29GB
NVMe Device
Total Memory:386 GB
2 sockets * 28 cores = 56 cores
Indexed Data Size: 272GB

--Regards,
Balmukund

8

Esrally starts a process per connection so am not sure how well it handles very large number of concurrent requests. As your data is larger than what can be cached you also need to look at disk I/O as this often is the bottleneck.

9

Hi Christian,
Thank you very much for your suggestions. I'll follow all your suggestions and will come back if there is any doubt.

--Regards,
Balmukund

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.