So, I've did more research and it seems the culprit is a query:
curl -XGET 'http://localhost:9200/dis_video/dis_video/_search?search_type=query_then_fetch' -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"query":"*testtttttttttttttttttt*"}}]}},{"bool":{"must_not":[{"term":{"status":{"value":-1,"boost":1}}}]}}]}},"from":0,"sort":{"id":{"order":"desc"},"_score":{"order":"desc"}},"size":0}'
Upon execution, I did follow the java heap memory in real time, which very soon reached 100% causing elasticsearch to crash and no output for the query.
heap.current heap.percent heap.max
7.9gb 100 7.9gb
I wasn't sure if this is because of the query, or the current elasticsearch version is having issues.
So I did upgrade elasticsearch from 6.8.21 to 7.17.7 and tried the same query, which was not executed actually, throwing this error:
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: field expansion for [*] matches too
many fields, limit: 1024, got: 58043","index_uuid":"iBH67VQTScWSCyDiFAU_7w","index":"dis_video"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"dis_video","node":"wLL8Vv76Sy-Fr3fJpGJDdg","reason":{"type":"query_shard_exception","reason":"failed to create query: field expansion for [*] matches too many fields, limit: 1024, got: 58043","index_uuid":"iBH67VQTScWSCyDiFAU_7w","index":"dis_video","caused_by":{"type":"illegal_argument_exception","reason":"field expansion for [*] matches too many fields, limit: 1024, got: 58043"}}}]},"status":400}
Seems like this newer version of Elasticsearch(7.17.7) knows the query is dangerous.
Googling for it, the very first result is the following:
"The default value is 1000. The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources"
If someone has the same issue, this research may be helpful.
