Elastic search 1000 req/s with max response time under 100 ms

Elastic Stack Elasticsearch
1

my server is 85 vCPU with 200 gb ram
i want to run 1000 req/s of medium search queries and aggregations with 100% success and max latency and response time under 100 ms
I need help on the configuration please for Elasticsearch the db is about 11 gb

2

only one index with 20 million records each one under 10 kb

3

What did you try so far? And what were your results.

Your entire 11G dataset fits (easily) in 200gb of RAM.

But I’ve no idea what a medium search query mean, is that line a medium steak?

”… and aggregations” is also very imprecise.

4

Your data is small enough to fit into a single primary shard as well as the page cache, which is good for the type of traffic volumes you describe.

I do have some further questions though.

Which version of Elasticsearch are you using?

Does you index have only a single primary shard?

How frequently do you update the data in the index?

What does your queries and mappings look like?

What latency do you see if you just send a single query at a time?

5

product example
{
"mongo_id": "671ccb8efa83bca45c9a6f74",
"id": "17258781",
"catalog_number": "MBS136332-1mg",
"supplier_catalog_number": "MBS136332",
"name": "Chicken Anti Rat Cotton IgG Polyclonal Affinity Purified HRP Labeled",
"size": "1mg",
"cluster_name": "purified-affinity-rabbit",
"supplier_id": "544",
"supplier_name": "MyBiosource",
"new_supplier_id": "544",
"new_supplier_name": "MyBiosource",
"buy_price": 0,
"sell_price": 0,
"price_valid_until_date": null,
"currency": "EUR",
"dry_ice": false,
"promotion_price": 0,
"filters": [
"Species:Rat",
"Host:Chicken",
"Isotype:IgG",
"Technique:Affinity",
"Technique:Polyclonal",
"Technique:Purified",
"Label:HRP"
]
}

20 million records
3 shards 1 replica
elastic 8.12.0

my current config
3 masters , 3 data, 3 coord

query example

{
"query": {
"bool": {
"filter": [
{ "term": { "name": "anti" } },
{ "term": { "filters": "Host:Chicken" } }
]
}
},
"size": 100,
"track_total_hits": false,
"aggs": { // Add this section back
"filters_count": {
"terms": {
"field": "filters",
"size": 250, // Reduced from 100 for better performance
"order": { "_count": "desc" } // More useful than ascending keys
}
}
}
}

6

product example
{
"mongo_id": "671ccb8efa83bca45c9a6f74",
"id": "17258781",
"catalog_number": "MBS136332-1mg",
"supplier_catalog_number": "MBS136332",
"name": "Chicken Anti Rat Cotton IgG Polyclonal Affinity Purified HRP Labeled",
"size": "1mg",
"cluster_name": "purified-affinity-rabbit",
"supplier_id": "544",
"supplier_name": "MyBiosource",
"new_supplier_id": "544",
"new_supplier_name": "MyBiosource",
"buy_price": 0,
"sell_price": 0,
"price_valid_until_date": null,
"currency": "EUR",
"dry_ice": false,
"promotion_price": 0,
"filters": [
"Species:Rat",
"Host:Chicken",
"Isotype:IgG",
"Technique:Affinity",
"Technique:Polyclonal",
"Technique:Purified",
"Label:HRP"
]
}

20 million records
3 shards 1 replica
elastic 8.12.0

my current config
3 masters , 3 data, 3 coord

query example

{
"query": {
"bool": {
"filter": [
{ "term": { "name": "anti" } },
{ "term": { "filters": "Host:Chicken" } }
]
}
},
"size": 100,
"track_total_hits": false,
"aggs": { // Add this section back
"filters_count": {
"terms": {
"field": "filters",
"size": 250, // Reduced from 100 for better performance
"order": { "_count": "desc" } // More useful than ascending keys
}
}
}
}