Heavily inspired by the Introducing Index Sorting in Elasticsearch 6.0 | Elastic Blog article I went to try index sorting out for faster AND conjunctions.
This is my index structure:
{
"index_name": {
"mappings": {
"dynamic": "false",
"properties": {
"brand_id": { "type": "keyword" },
"category_id": { "type": "keyword" },
"status_id": { "type": "keyword" },
"user_id": { "type": "keyword" },
"price": { "type": "scaled_float", "scaling_factor": 100.0 },
"created_at": { "type": "date" }
}
},
"settings": {
"index": {
"sort": {
"field": ["category_id", "brand_id", "status_id", "created_at"],
"order": ["asc", "asc", "asc", "desc"]
}
}
}
}
}
And this is the actual query:
{
"_source": false,
"size": 24,
"query": {
"bool": {
"filter": [
{
"term": {
"category_id": 197
}
},
{
"term": {
"brand_id": 51
}
},
{
"term": {
"status_id": 2
}
},
{
"range": {
"price": {
"gte": 2.0,
"lte": 5.0
}
}
}
],
"must_not": {
"terms": {
"user_id": [12345]
}
}
}
},
"sort": {
"created_at": "desc"
}
}
The results have been mixed.
Some performed great and their performance has skyrocketed, however, some became completely horrible and started throwing the following exceptions:
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "topDocs already consumed"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "index_name",
"node": "lx9S3vwES4y2n2xa-PFyUw",
"reason": {
"type": "illegal_state_exception",
"reason": "topDocs already consumed"
}
}
],
"caused_by": {
"type": "illegal_state_exception",
"reason": "topDocs already consumed",
"caused_by": {
"type": "illegal_state_exception",
"reason": "topDocs already consumed"
}
}
},
"status": 500
}
This surely must be related to index sorting but I couldn't find anything indicating why this could be raised.
- The index contains 100M documents and weights 10G
- Elasticsearch 7.12.0
- Using built-in JVM
- Running on CentOS servers