Hi there,
I was attempting to use completion suggersters functionality on an email field. My mapping included this:
"email": {
"type": "keyword",
"fields": {
"completion": {
"type": "completion"
}
}
}
This was run on AWS Elasticsearch service with 6 instances of type r4.xlarge. After indexing around 2.5 Billion emails, the Elasticsearch cluster started blocking requests as the JVM Memory Pressure exceeded over 90%. I stopped any more documents from being indexed, and didn't perform any searches on the cluster, however the memory pressure stayed consistently high. (NOTE: This email index is the only index in the cluster)
After spending some time trying to understanding why this occured, the only sound reason I came up with was that this is due to the completion suggester. The little that I know about 'completion suggester' is that it uses an in-memory data structure, so maybe after a certain number (2.5 billion docs + ), the auto-completion doesn't work as its too taxing on the RAM.
Would be great to get someones input whether my conclusion is right. And how the internals of completion suggesters work. And if you think it isn't due to the completion suggester, then what are some other possible reasons for the memory pressure staying consistently over 90%?
Thanks