For development purposes we set up an elastic cluster on a virtual machine within our own data center consisting of two nodes with 4 GB RAM each, running on version 6.2.3.
Our elastic cloud cluster (cloud.elastic.co) for production also consists of two nodes with 8 GB RAM each running on version 6.3.2.
Usually searches on the cloud cluster are a bit faster than those on our self hosted machines. At least the ones we used until now (mostly simple field matches).
However I'm currently developing an index for autocomplete searches. Turns out, the same index, with exactly the same settings, mappings and data is up to 100 times slower on our prod cluster than on our dev cluster!
Settings:
{
"autocomplete_b": {
"settings": {
"index": {
"refresh_interval": "1s",
"number_of_shards": "5",
"provided_name": "autocomplete_b",
"creation_date": "1537864159701",
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": "1",
"max_gram": "20"
}
},
"analyzer": {
"lowercase_ascii_analyzer": {
"filter": [
"lowercase",
"asciifolding"
],
"type": "custom",
"tokenizer": "standard"
},
"autocomplete_analyzer": {
"filter": [
"lowercase",
"asciifolding",
"autocomplete_filter"
],
"type": "custom",
"tokenizer": "standard"
}
}
},
"number_of_replicas": "1",
"uuid": "Q6S9CQTSTqOdg0FLwzPZWg",
"version": {
"created": "6020399"
}
}
}
}
}
Mapping:
{
"autocomplete_b": {
"mappings": {
"autocompleteitem": {
"properties": {
"countryCodeIsoA3": {
"type": "keyword"
},
"destinationType": {
"type": "integer"
},
"hotelId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"hotelMasterId": {
"type": "integer"
},
"hrsLocationId": {
"type": "integer"
},
"hrsParentLocationId": {
"type": "integer"
},
"id": {
"type": "keyword"
},
"latitude": {
"type": "double"
},
"longitude": {
"type": "double"
},
"numberOfHotels": {
"type": "integer"
},
"translations": {
"properties": {
"BG": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"CZ": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"DA": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"DE": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"EN": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"ES": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"FI": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"FR": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"HU": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"IT": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"JA": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"KO": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"NB": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"NL": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"PL": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"PT": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"RU": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"SV": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"TR": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"ZH": {
"type": "text",
"analyzer": "autocomplete_analyzer"
}
}
}
}
}
}
}
}
Here's a sample query:
GET autocomplete/_search
{
"_source": {
"includes": [
"id", "translations.EN", "countryCodeIsoA3", "numberOfHotels", "destinationType", "hrsLocationId", "hotelId", "latitude", "longitude"
]
},
"query": {
"match": {
"translations.EN": {
"query": "Seattle Hotel",
"operator": "and",
"analyzer": "lowercase_ascii_analyzer"
}
}
},
"size": 5
}
The result from our development cluster: 9ms would already be fantastic, but subsequent requests with the same search string return in 2ms consistently.
However the same request on our elastic cloud cluster takes 493ms (54 times slower). Often the difference is even a lot bigger. The processing time of subsequent requests varies extremely: 310, 499, 354, 244, 4, 178, 382
(BTW: Our index contains a lot of documents, currently 1'656'885.)
To repeat myself: The indices on development and production are set up 100% the same way, with exactly the same data.
Does anyone have a clue where this inconsistent behavior could come from?